Output control apparatus, display terminal, remote control system, control method, and non-transitory computer-readable medium

ABSTRACT

An output control apparatus is communicable with a communication apparatus through a communication network. The communication apparatus includes a first image capturing device configured to capture a subject at a remote site to acquire a first image and a second image capturing device configure to capture a part of the subject to acquire a second image. The output control apparatus includes circuitry to: receive the first image transmitted from the communication apparatus; output the received first image so as to be displayed on a display; receive, from the communication apparatus, the second image acquired by capturing a part of the subject corresponding to a display position of the first image displayed on the display; output the received second image so as to be displayed on the display; and control the display to display the first image and the second image that are output.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application Nos. 2019-141212, filedon Jul. 31, 2019, and 2019-141328, filed on Jul. 31, 2019, in the JapanPatent Office, the entire disclosures of which are hereby incorporatedby reference herein.

BACKGROUND Technical Field

The present disclosure relates to an output control apparatus, a displayterminal, a remote control system, a control method, and anon-transitory computer readable medium.

Description of the Related Art

A remote control system is known in which a telepresence robot(hereinafter, referred to as a “robot” in order to simplify thedescription) provided at a remote site is remotely controlled using adisplay terminal located at a different site through a communicationnetwork. This remote control system controls the display terminal todisplay an image captured by an image capturing device provided at therobot, whereby allowing a user to check information of the site wherethe robot is provided from a remote location.

Another technique is known that uses a robot provided with differenttypes of image capturing devices, to allow a user to check surroundingsof the robot in more detail. For example, a radio-controlled mobileobject is known that includes a front camera and a rear camera thatimages the front and the rear of a traveling apparatus and an overheadcamera that can image the entire surroundings around the mobile object.

SUMMARY

According to an embodiment, an output control apparatus is communicablewith a communication apparatus through a communication network. Thecommunication apparatus includes a first image capturing deviceconfigured to capture a subject at a remote site to acquire a firstimage and a second image capturing device configure to capture a part ofthe subject to acquire a second image. The output control apparatusincludes circuitry to: receive the first image transmitted from thecommunication apparatus; output the received first image so as to bedisplayed on a display; receive, from the communication apparatus, thesecond image acquired by capturing a part of the subject correspondingto a display position of the first image displayed on the display;output the received second image so as to be displayed on the display;and control the display to display the first image and the second imagethat are output.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof may be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 is a diagram illustrating an example of a system configuration ofa remote control system, according to an embodiment of the presentdisclosure;

FIG. 2 is a diagram illustrating an example of a schematic configurationof a robot, according to an embodiment of the present disclosure;

FIG. 3 is a diagram illustrating a variation 1 (1-1) of a schematicconfiguration of the robot, according to an embodiment of the presentdisclosure;

FIG. 4A, FIG. 4B and FIG. 4C are diagrams illustrating a variation 1(1-2) of a schematic configuration of the robot, according to anembodiment of the present disclosure;

FIG. 5A and FIG. 5B are diagrams illustrating a variation 2 (2-1) of aschematic configuration of the robot, according to an embodiment of thepresent disclosure;

FIG. 6 is a diagram illustrating a variation 2 (2-2) of a schematicconfiguration of the robot, according to an embodiment of the presentdisclosure;

FIG. 7A, FIG. 7B and FIG. 7C are diagrams illustrating a variation 2(2-3) of a schematic configuration of the robot, according to anembodiment of the present disclosure;

FIG. 8A and FIG. 8B are diagrams illustrating a variation 2 (2-4) of aschematic configuration of the robot, according to an embodiment of thepresent disclosure;

FIG. 9A and FIG. 9B are diagrams illustrating a variation 2 (2-5) of aschematic configuration of the robot, according to an embodiment of thepresent disclosure;

FIG. 10A, FIG. 10B and FIG. 10C are diagrams illustrating a variation 3of a schematic configuration of the robot, according to an embodiment ofthe present disclosure;

FIG. 11 is a block diagram illustrating an example of a hardwareconfiguration of the robot, according to an embodiment of the presentdisclosure;

FIG. 12 is a block diagram illustrating an example of a hardwareconfiguration of a display terminal, according to according to anembodiment of the present disclosure;

FIG. 13 is a block diagram illustrating an example of a hardwareconfiguration of a communication management server, according to anembodiment of the present disclosure;

FIG. 14 is a block diagram illustrating an example of a hardwareconfiguration of a special image capturing device, according to anembodiment of the present disclosure;

FIG. 15A is an illustration of a hemispherical image (front side)captured by the special image capturing device, according to anembodiment of the present disclosure;

FIG. 15B is an illustration of a hemispherical image (back side)captured by the special image capturing device, according to anembodiment of the present disclosure;

FIG. 15C is an illustration of an image represented by equirectangularprojection, according to an embodiment of the present disclosure;

FIG. 16A is a conceptual diagram illustrating an example of how anequirectangular projection image maps to a surface of a sphere,according to an embodiment of the present disclosure;

FIG. 16B is an illustration of a spherical image, according to anembodiment of the present disclosure;

FIG. 17 is an illustration of relative positions of a virtual camera anda viewable area in a case where the spherical image is represented as asurface area of a three-dimensional solid sphere, according to anembodiment of the present disclosure;

FIG. 18A is a perspective view of FIG. 17, according to an embodiment ofthe present disclosure;

FIG. 18B is an illustration of an image of the viewable area displayedon a display of the display terminal, according to an embodiment of thepresent disclosure;

FIG. 19 is a view illustrating a relation between viewable-areainformation and the image of the viewable area, according to anembodiment of the present disclosure;

FIG. 20 is a block diagram illustrating an example of a functionalconfiguration of the remote control system, according to an embodimentof the present disclosure;

FIG. 21A is a conceptual diagram illustrating an example of a commandmanagement table, according to an embodiment of the present disclosure;

FIG. 21B is a conceptual diagram illustrating an example of an imagecapturing parameter management table, according to an embodiment of thepresent disclosure;

FIG. 22A is a conceptual diagram illustrating an example of a stateinformation management table, according to an embodiment of the presentdisclosure;

FIG. 22B is a conceptual diagram illustrating an example of a conditioninformation management table, according to an embodiment of the presentdisclosure;

FIG. 23A is a conceptual diagram illustrating an example of a usercommand management table, according to an embodiment of the presentdisclosure;

FIG. 23B is a conceptual diagram illustrating an example of a displaystate management table, according to an embodiment of the presentdisclosure;

FIG. 24A is a conceptual diagram illustrating an example of anauthentication management table, according to an embodiment of thepresent disclosure;

FIG. 24B is a conceptual diagram illustrating an example of a terminalmanagement table, according to an embodiment of the present disclosure;

FIG. 25A is a conceptual diagram illustrating an example of adestination list management table, according to an embodiment of thepresent disclosure;

FIG. 25B is a conceptual diagram illustrating an example of a sessionmanagement table, according to an embodiment of the present disclosure;

FIG. 26 is a sequence diagram illustrating an example of operationperformed at preparatory stage for starting data exchange between therobot and the display terminal in the remote control system, accordingto an embodiment of the present disclosure;

FIG. 27 is a diagram illustrating an example of a destination listscreen displayed on the display terminal, according to an embodiment ofthe present disclosure;

FIG. 28 is a sequence diagram illustrating an example of operation froma selection of a destination candidate to a start of datatransmission/reception, performed by the remote control system,according to an embodiment of the present disclosure;

FIG. 29 is a sequence diagram illustrating an example of operation oftransmitting various data from the robot to the display terminal in theremote control system, according to an embodiment of the presentdisclosure;

FIG. 30 is a diagram illustrating an example of a display screendisplayed on the display terminal, according to an embodiment of thepresent disclosure;

FIG. 31 is a diagram illustrating an example of state informationindicating a state of the robot, according to an embodiment of thepresent disclosure;

FIG. 32 is an illustration of an example of a display screen displayedon the display terminal when the robot is moving forward in the remotecontrol system, according to an embodiment of the present disclosure;

FIG. 33 is a diagram for describing an example of the spherical imagedisplayed on the display terminal, according to an embodiment of thepresent disclosure;

FIG. 34 is a flowchart illustrating an example of operation ofcontrolling the robot based on a movement state of the robot performedby the display terminal, according to an embodiment of the presentdisclosure;

FIG. 35 is a flowchart illustrating an example of operation ofcontrolling the robot based on an input command, performed by thedisplay terminal, according to an embodiment of the present disclosure;

FIG. 36 is a flowchart illustrating an example of operation ofcontrolling robot based on a request command from the display terminal,performed by the robot, according to an embodiment;

FIG. 37 is a flowchart illustrating an example of operation of changinga display position of the spherical image on the display terminal,according to an embodiment of the present disclosure;

FIG. 38 is a sequence diagram illustrating an example of an operation ofchanging an imaging position, performed by a generic image capturingdevice, according to an embodiment of the present disclosure;

FIG. 39A is a diagram illustrating an example of the spherical imagedisplayed on the display terminal, according to an embodiment of thepresent disclosure;

FIG. 39B is a diagram illustrating an example of a detailed imagedisplayed on the display terminal, according to an embodiment of thepresent disclosure;

FIG. 40 is a diagram illustrating an example of the spherical imagedisplayed on the display terminal, according to an embodiment of thepresent disclosure;

FIG. 41 is a diagram illustrating an example of a display screendisplayed on a head-mounted display as an example of the displayterminal, according to an embodiment of the present disclosure;

FIG. 42 is a sequence diagram illustrating an example of operation ofcausing the robot to display a captured image acquired by the displayterminal, performed by the remote control system, according to anembodiment of the present disclosure;

FIG. 43A and FIG. 43B are diagrams each illustrating an example of adisplay screen displayed on the robot, according to an embodiment of thepresent disclosure;

FIG. 44 is a diagram illustrating an example of a system configurationof a remote control system, according to a variation of an embodiment ofthe present disclosure;

FIG. 45 is a diagram illustrating an example of a functionalconfiguration of the remote control system, according to a variation ofan embodiment of the present disclosure;

FIG. 46 is a sequence diagram illustrating an example of operation oftransmitting various data from the robot to the display terminal in theremote control system, according to a variation of an embodiment of thepresent disclosure; and

FIG. 47 is a flowchart illustrating an example of an operation ofperforming image processing on the display screen data, performed by aninformation processing server, according to a variation of an embodimentof the present disclosure.

The accompanying drawings are intended to depict embodiments of thepresent disclosure and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentdisclosure. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise.

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Embodiments of the present disclosure are described with reference tothe drawings. In the description of the drawings, the same elements aredenoted by the same reference numerals, and redundant descriptionsthereof are omitted.

System Configuration:

FIG. 1 is a diagram illustrating an example of a system configuration ofa remote control system, according to an embodiment. The remote controlsystem illustrated in FIG. 1 allows robots 10 located at different sitesand a display terminal 50 used by an administrator located at a remoteplace to perform remote communication, to perform management ormaintenance of devices of apparatuses in the sites or check the positionor movement route of persons who are in the sites.

The remote control system 1 a includes robots 10A, 10B, 10C respectivelylocated at a plurality of sites (site A, site B, site C), a displayterminal 50, and a communication management server 90. The robots 10A,10B, and 10C are collectively referred to as a “robot 10” or “robots10”, unless they need to be distinguished from each other. The robots10, the display terminal 50, and the communication management server 90are communicably connected through a communication network 9. Thecommunication network 9 is implemented by, for example, a local areanetwork (LAN), a dedicated line, the Internet, etc. The communicationnetwork 9 may not only include a wired network, but also a wirelessnetwork such as a network in compliance with Wi-Fi (registeredtrademark) or the like.

The robots 10 are mobile objects that are respectively provided in thesites (site A, site B, site C) and autonomously travel in accordancewith remote control from the display terminal 50. Each of the robots 10moves in the site while imaging subjects in a wide range around therobot 10 by a special image capturing device 21 described below, andtransmits a spherical image acquired by the special image capturingdevice 21 to the display terminal 50, whereby providing information(image) in the site to an operator who operates the robot 10 using thedisplay terminal 50. Further, each of the robots 10 captures a part ofthe subjects captured by the special image capturing device 21 by usinga generic image capturing device 24 described below, and transmits adetailed image acquired by the generic image capturing device 24 to thedisplay terminal 50, whereby providing detailed information (image) of aparticular area in the site to the operator who operates the robot 10using the display terminal 50. The robot 10 is an example of a mobileobject.

The display terminal 50 is a terminal apparatus such as a personalcomputer (PC) configured to remotely control the robot 10 provided ateach of the sites (site A, site B, site C). The display terminal 50displays the spherical image or the detailed image transmitted from therobot 10. The operator can remotely operate the robot 10 while viewingthe image displayed on the display terminal 50. The display terminal 50is an example of an output control apparatus.

The display terminal 50 is any suitable apparatus, provided that itincludes display means configured to display the image transmitted fromthe robot 10. Examples of the display terminal 50 include, but notlimited to, a tablet terminal, a mobile phone, a smartphone, a wearableterminal such as a head-mounted display (HMD), a communication terminalprovided with a wide-angle screen (cylindrical, spherical,semi-spherical screen, etc.), and a personal digital assistant (PDA).

The communication management server 90 is a server computer configuredto manage communication between each of the robots 10 located atdifferent sites and the display terminal 50. The communicationmanagement server 90 is connected to each of the robots 10 and thedisplay terminal 50 through the communication network 9. In one example,the communication management server 90 is configured as a singlecomputer. In another example, the communication management server 90 isconfigured as a plurality of computers to which divided portions(functions, means, or storages) are arbitrarily allocated.

Examples of the site where the robot 10 is provided include, but notlimited to, an office, a school, a warehouse, a factory, and aconstruction site. The operator who operates the robot 10 by using thedisplay terminal 50 checks the image of the site transmitted from therobot 10, to recognize the position or movement route of a person who ispresent in the site and to perform management and maintenance of anapparatus provided in the site. Further, the robot 10 and the displayterminal 50 exchange images captured by both of them, to performbidirectional communication (remote conference).

Although in FIG. 1, the description is given of the configuration inwhich one robot 10 is provided in each site, no limitation is intendedthereby. In another example, plural robots 10 are provided in one site.Further, in one example, the display terminal 50 is configured tocommunicate with each of the robots 10 provided at plural sites. Inanother example, the display terminal 50 is configured to communicateonly with the robot 10 provided in one site.

Configuration of Robot:

A description is now given of a detailed configuration of the robot 10illustrated in FIG. 1, with reference to FIG. 2 to FIG. 10A, FIG. 10B,and FIG. 10C. FIG. 2 is a diagram illustrating an example of a schematicconfiguration of the robot, according to an embodiment.

The robot 10 a illustrated in FIG. 2 includes the special imagecapturing device 21, a housing 15 including a control device 30configured to control the processing or operation of the robot 10 a, amovement mechanism 17 a and a movement mechanism 17 b, each beingconfigured to move the robot 10 a, and a support 13. The movementmechanism 17 a and the movement mechanism 17 b are hereinaftercollectively referred to as a “movement mechanism 17”, unless they needto be distinguished from each other.

The special image capturing device 21 is an image capturing deviceconfigured to photograph a subject such as a person, an object, and alandscape to obtain a spherical (360 degrees) panoramic image. Thespecial image capturing device 21 is a special digital camera, whichcaptures an image of a subject to obtain two hemispherical images, fromwhich a spherical (panoramic) image is generated. A detailed descriptionis given later of the spherical image captured by the special imagecapturing device 21 and the hemispherical images, from which thespherical image is generated, with reference to FIG. 15A, FIG. 15B, FIG.15C to FIG. 19. The special image capturing device 21 is an example ofimage capturing means (first image capturing means) for capturing animage of a subject in a remote site.

The robot 10 a transmits spherical image data, which is data of thespherical image acquired by the special image capturing device 21, tothe display terminal 50. The image of the spherical image data is amoving image (video or a still image, or both of the moving image(video) and the still image. Further, the spherical image data mayinclude sound data as well as image data.

The spherical image is one example of an image acquired by the specialimage capturing device 21. Another example of the image acquired by thespecial image capturing device 21 is a wide-angle image having an angleof view of a predetermined value or more. In this case, the wide-angleimage is acquired by a wide-angle camera, a stereo camera, or the like.In other words, the special image capturing device 21 is image capturingmeans configured to acquire an image, such as a spherical image and awide-angle image, photographed by using a lens having a focal lengthshorter than a predetermined value. The image (the spherical image orthe wide-angle image) acquired by the special image capturing device 21is an example of a first image. In the following embodiments, adescription is given of an example in which an image acquired by thespecial image capturing device 21 is a spherical image.

The housing 15 is provided in the body of the robot 10 a. The housing 15includes a power supply unit that supplies necessary power to the robot10 a in its entirety and the control device 30 that controls theprocessing or operation of the robot 10 a, which are built in thehousing 15.

The support 13 is a member that mounts (fixes) the special imagecapturing device 21 on the robot 10 a (housing 15). Examples of thesupport 13 include, but not limited to, a pole or the like fixed to thehousing 15, and a base fixed to the housing 15. In another example, thesupport 13 is a movable member configured to adjust the imagingdirection (orientation) and position (height) of the special imagecapturing device 21. In the robot 10 a illustrated in FIG. 2, since thespecial image capturing device 21 is fixed by the support 13, thespecial image capturing device 21 always faces the same direction as thedrive direction of the robot 10 a. This makes it easy for the operatorwho operates the robot 10 to perform the operation while viewing thespherical image acquired by the special image capturing device 21.

The movement mechanism 17 is a unit that causes the robot 10 a to move,and includes, but not limited to, wheels, a traveling motor, a travelingencoder, a steering motor, and a steering encoder. Since the movementcontrol of the robot 10 a is an existing technique, detailed descriptionthereof is omitted herein. The robot 10 a receives a travelinginstruction from the operator (display terminal 50), and the movementmechanism 17 moves the robot 10 based on the received travelinginstruction.

In the embodiment, a description is given of an example in which themovement mechanism 17 includes two wheels. In another example, themovement mechanism 17 is a bipedal type or a single wheel type. Inaddition, the shape of the robot 10 a is not limited to the vehicle typeas illustrated in FIG. 2. In another example, the robot 10 a is abipedal humanoid type, or has a shape imitating a living thing, a shapeimitating a particular character, or the like.

Variations of Robot Configuration:

Variation 1:

A description is now given of Variation 1 of the configuration of therobot 10, with reference to FIG. 3 and FIG. 4A to FIG. 4C. The robot 10b (10 b 1 to 10 b 4) illustrated in FIG. 3 and FIG. 4A to 4C includes,in addition to the configuration of the robot 10 a illustrated in FIG.2, the generic image capturing device 24 that is movable and photographsa part of the subject photographed by the special image capturing device21 to acquire a detailed image. Such generic image capturing device 24that is movable is hereinafter referred to as a “movable camera 23”. Themovable camera 23 is one type of example of the generic image capturingdevice 24. The movable camera 23 illustrated in FIG. 3 and FIG. 4A to 4Cis a movable generic image capturing device that includes a built-inmovement mechanism, and is configured to acquire an image in a desireddirection and to zoom-in and zoom-out. In the robot 10 b 1 illustratedin FIG. 3, the movable camera 23 is mounted on the housing 15.

In the disclosure, the generic image capturing device 24 is a digitalcamera configured to acquire a flat image (detailed image), such as adigital single-lens reflex camera and a compact digital camera. Thegeneric image capturing device 24 is an example of second imagecapturing means. The robot 10 b transmits data of the detailed imageacquired by the generic image capturing device 24 (movable camera 23) tothe display terminal 50. The detailed image acquired by the genericimage capturing device 24 is an image acquired by photographing a partof the object photographed by the special image capturing device 21 atan angle of view of a predetermined value or more. In other words, thegeneric image capturing device 24 is image capturing means configured toacquire an image (detailed image) photographed using a lens having alonger focal length than the lens of the special image capturing device21. The image (detailed image, flat image) acquired by the generic imagecapturing device 24 is an example of a second image.

The display terminal 50 displays the spherical image that allows theoperator of the robot 10 b to view the surroundings of the robot 10 b ina wide range, when the robot 10 b is moved by remote control by theoperator, for example. Further, the display terminal 50 displays thedetailed image acquired by the generic image capturing device 24 whenthe operator of the robot 10 b wants to check detailed information of aparticular area included in the spherical image. In other words, thespecial image capturing device 21 is an example of image capturing meansconfigured to perform a photographing process to acquire an image (thespherical image or the wide-angle image) that allows the operator of therobot 10 to check the surroundings around the robot 10 b in a widerange. The generic image capturing device 24 is an example of imagecapturing means configured to perform a photographing process to acquirean image (the detailed image) that allows the operator of the robot 10 bto check the state of a specific part around the robot 10 b in detail.This allows the operator of the robot 10 b to select the spherical imageor the detailed image to be displayed on the display terminal 50according to applications. Thus, the operability of the operator of therobot 10 is improved.

The robot 10 b 2 illustrated in FIG. 4A includes the movable camera 23mounted downward on a mount member 18, which is fixed to the support 13.There is a blind spot where the movable camera 23 cannot photograph anobject. In the case of the robot 10 b 1 illustrated in FIG. 3, thedownward direction of the robot 10 b 1 is a blind spot of the movablecamera 23. This makes it difficult for the operator to recognizeobstacles when moving the robot 10 b 1. To address such issue, themovable camera 23 is mounted on the robot 10 b 2 downward to eliminatethe downward blind spot. Further, the robot 10 b 3 illustrated in FIG.4B includes a movable camera 23 a mounted upward and a movable camera 23b mounted downward on the mount member 18 fixed to the support 13. Withsuch configuration, the robot 10 b 3 photographs both the upwarddirection and the downward direction of the robot 10 b 2 using themovable camera 23 a and the movable camera 23 b, whereby furtherreducing the blind spot of the imaging position.

Further, in the robot 10 b 3 illustrated in FIG. 4C, the special imagecapturing device 21 is mounted downward on a mount member 18 a fixed tothe support 13, and the movable camera 23 is mounted upward on a mountmember 18 b fixed to the support 13. Thus, in the robot 10 b 3, theimage capturing position of the special image capturing device 21 andthe imaging position of the movable camera 23 are close to each other.This enables to minimize the difference between the visual fields of thespecial image capturing device 21 and the movable camera 23.

Variation 2:

A description is now given of Variation 2 of the configuration of therobot 10, with reference to FIG. 5A and FIG. 5B to FIG. 9A and FIG. 9B.The robot 10 c (10 c 1 to 10 c 5) illustrated in FIG. 5A and FIG. 5B toFIG. 9A and FIG. 9B includes, in addition to the configuration of therobot 10 a illustrated in FIG. 2, the generic image capturing device 24that is fixed type and photographs a part of the subject photographed bythe special image capturing device 21 to acquire a detailed image. Suchgeneric image capturing device 24 that is fixed type is hereinafterreferred to as a “normal camera 25”. The normal camera 25 is one type ofthe generic image capturing device 24.

The robot 10 c 1 illustrated in FIG. 5A includes the normal camera 25mounted on the mount member 18, which is fixed to the support 13. Inthis case, unlike the configuration of Variation 1 in which the movablecamera 23 is provided, the robot 10 c 1 does not photograph a subject ina desired direction. However, the robot 10 c 1 acquires a detailed imageof a subject in the front of the robot 10 c 1. Further, the robot 10 c 2illustrated in FIG. 5B uses a movable arm 11 to adjust the imagingposition (imaging direction) of the normal camera 25 mounted on themount member 18. The movable arm 11 rotates by using a rotation shaft 12to change the direction of the normal camera 25. The robot 10 c 2changes the imaging position of the normal camera 25 by rotating ordeforming the movable arm 11 as well as by changing the direction of therobot 10 by the movement mechanism 17. Furthermore, the robot 10 c 3illustrated in FIG. 6 includes, in addition to the configuration of therobot 10 b 2 illustrated in FIG. 5B, the special image capturing device21 mounted on the mount member 18. This configuration enables the robot10 b 3 to use the movable arm 11 to change the imaging position of thespecial image capturing device 21 as well as the normal camera 25.

The robot 10 illustrated in FIG. 7A to FIG. 7C has a differentconfiguration illustrated in FIG. 6 as to the arrangement of the specialimage capturing device 21. Since the special image capturing device 21and the normal camera 25 have different photographing purposes asdescribed with reference to FIG. 6, it is preferable to change thearrangement of the special image capturing device 21 and the normalcamera 25 depending on usage.

The robot 10 illustrated in FIG. 7A includes the special image capturingdevice 21 arranged on the normal camera 25. The special image capturingdevice 21 is required to photograph a wide area around the robot 10.Therefore, as the robot 10 has the arrangement as illustrated in FIG.7A, the effect of properly using the special image capturing device 21and the normal camera 25 becomes more pronounced.

Further, the robot 10 illustrated in FIG. 7B includes the special imagecapturing device 21 arranged behind the normal camera 25. An area in thefront direction (imaging direction) of the normal camera 25 is an areawhich the operator of the robot 10 wants to check in detail. Therefore,the arrangement as illustrated in FIG. 7B enables the normal camera 25to photograph an area in the front direction (imaging direction) withoutbeing obstructed by obstacles or the like. Furthermore, the arranging asillustrated in FIG. 7B enables the special image capturing device 21 tophotograph an area that cannot be photographed by the normal camera 25(e.g., a region behind the normal camera 25) with a relatively goodresolution, without the normal camera 25 and the like being reflected.

Moreover, the robot 10 illustrated in FIG. 7C includes the special imagecapturing device 21 arranged below the normal camera 25. The conditionof the ground (foot) is important when the robot 10 moves. Therefore,the arrangement as illustrated in FIG. 7C enables the special imagecapturing device 21 to photograph the ground (foot) without beingdisturbed by the normal camera 25 or the mount member 18. This allowsthe operator of the robot 10 to move the robot 10 more safely whileviewing the spherical image captured by the special image capturingdevice 21.

The robot 10 illustrated in FIG. 8A and FIG. 8B has a differentconfiguration from that of FIG. 6 in the structure of the movable arm11. It is preferable that the movable arm 11 have movable range requireddepending on usage of the robot 10. The movable arm 11 a illustrated inFIG. 8A includes no joint member and only changes the direction thereofby the rotation shaft 12. Such configuration of the robot 10 suffices aslong as the height and distance of the portion to be photographed by thespecial image capturing device 21 or the normal camera 25 are constant.The movable arm 11 b illustrated in FIG. 8B includes a joint memberconfigured to deform the movable arm 11 a illustrated in FIG. 8A. Inthis case, the movable arm 11 b is vertically deformed.

The robot 10 c 4 illustrated in FIG. 9A includes an expansion member 14configured to expand and contract the support 13 of the robot 10 c 2illustrated in FIG. 5B. The robot 10 c 4 adjusts the height of thespecial image capturing device 21 by expanding and contracting thesupport 13 using the expansion member 14. This allows the robot 10 c 4to perform flexible processing. For example, by extending the support13, the robot 10 c 4 enables the special image capturing device 21 tophotograph a subject that exists ahead of an obstacle existing aroundthe robot 10 c 4. Further, for example, by shrinking the support 13, therobot 10 c 4 moves while photographing the state of the ground (foot) bythe special image capturing device 21.

The robot 10 c 5 illustrated in FIG. 9B has a configuration in which thespecial image capturing device 21 and the normal camera 25 are mountedon different movable arms 11 respectively. With this configuration, therobot 10 c 5 enables the special image capturing device 21 to performphotographing from an appropriate position by deforming the movable arm11 on which the special image capturing device 21 is mounted and enablesthe normal camera 25 to photograph a portion to be checked in moredetail by deforming the movable arm 11 on which the normal camera 25 ismounted.

Variation 3:

A description is now given of Variation 3 of the configuration of therobot 10, with reference to FIG. 10A to FIG. 10C. The robot 10 d (10 d 1to 10 d 3) as illustrated in FIG. 10A to FIG. 10C includes a display 150configured to display information on the display terminal 50 in additionto the configuration of the robots 10 a to 10 c as illustrated in FIG. 2to FIG. 9A and FIG. 9B. The display 150 is arranged in the robot 10 d 1illustrated in FIG. 10A so that the display screen can be checked fromthe front of the robot 10 d 1. The display 150 displays a captured imageobtained by photographing the operator who is in the site where thedisplay terminal 50 is provided. Further, on the display 150,information indicating where the operator is actually looking isdisplayed together with the image of the operator. Thus, the robot 10 d1 enables the user who is in the site where the robot 10 d 1 is locatedto recognize the direction in which the operator at the remote locationis looking.

The robot 10 d 2 illustrated in FIG. 10B includes a display 150 a at thefront side of the robot 10 d 2 so that the display screen can be checkedfrom the front of the robot 10 d 2. Further, the robot 10 d 2 includes adisplay 150 b at the rear side of the robot 10 d 2 so that the displayscreen can be checked from behind the robot 10 d 2. FIG. 10B illustratesthe robot 10 d 2 viewed from the side. Thus, as the robot 10 d 2includes plural displays 150 symmetrically at the front side and therear side of the robot 10 d 2, also a user behind the robot 10 d 2recognizes the direction in which the operator is looking. An image ofthe back of the operator's head, which image is prepared in advance, isdisplayed on the display 150 a or the display 150 b provided rearwardlyrelative to the direction in which the operator is looking. Although thedescription given above is of an example in which the robot 10 d 2includes two displays, i.e., the display 150 a and the display 150 b,even the single display 150 displays the same or substantially the sameimages, when the display 150 is implemented by a convex display, acollapsible display, an omnidirectional display, or the like arrangedaround the robot 10 d 2. In this case, the user around the robot 10 d 2can check the display screen from the side of the robot 10 d 2, forexample.

Further, the robot 10 d 3 illustrated in FIG. 10C includes an indicatorlamps 160 all around the housing 15. The indicator lamps 160 display thedrive direction of the robot 10 d 3. FIG. 10C illustrates the robot 10 d3 viewed from the side. The robot 10 d 3 turns on at least one of theindicator lamps 160, the at least one being provided at a positioncorresponding to the display position of the spherical image displayedon the display terminal 50 and the imaging position of the generic imagecapturing device 24. In another example, the robot 10 d 3 turns on atleast one of the indicator lamps 160 in different colors or brightness,to distinguish the display position of the spherical image and theimaging position of the generic image capturing device 24. Thus, therobot 10 d 3 allows the user who is around the robot 10 d 3 to recognizethe direction in which the operator is looking, depending on thelighting state of the indicator lamps 160. Although the descriptiongiven above is of an example in which the indicator lamps 160 arearranged around the housing 15, the embodiment is not limited thereto.For example, the indicator lamps 160 are arranged in any suitableposition, provided that they are arranged around the robot 10 d 3. Thedisplay 150 (150 a, 150 b) and the indicator lamps 160 of the robot 10 das illustrated in FIG. 10A to FIG. 10C are examples of display meansconfigured to indicate the direction in which the operator of the robot10 (the user of the display terminal 50) is looking at the sphericalimage.

In another example, the robot 10 described above with reference to FIG.2 to FIG. 10A to FIG. 10C includes various sensors configured to detectinformation around the robot 10 in addition to the above-describedconfiguration. Examples of the various sensors include sensor devicessuch as a barometer, a thermometer, a photometer, a human sensor, and anilluminance meter. In still another example, the robot 10 illustrated inFIG. 2 to FIG. 10A to FIG. 10c includes operation means configured toallow the robot 10 to perform other operations in addition to moving.Examples of the operation means include, but not limited to, a robothand configured to grasp an object. In still another example, the robot10 includes a single image capturing device that implements thefunctions of the special image capturing device 21 and the generic imagecapturing device 24.

Hardware Configuration:

A description is now given of a hardware configuration of eachapparatus, device, or terminal of the remote control system 1 a, withreference to FIG. 11 to FIG. 14. In the hardware configurationillustrated in FIG. 11 to FIG. 14, components or elements may be addedor deleted as needed.

Hardware Configuration of Robot:

FIG. 11 is a block diagram illustrating an example of a hardwareconfiguration of the robot 10, according to an embodiment. The robot 10includes the control device 30 that controls processing or operation ofthe robot 10. The control device 30 is provided inside the housing 15 ofthe robot 10, as described above. In another example, the control device30 is provided outside the housing 15 of the robot 10, or is provided asa separate device from the robot 10.

The control device 30 includes a central processing unit (CPU) 301, aread only memory (ROM) 302, a random access memory (RAM) 303, a harddisk drive (HDD) 304, a medium interface (I/F) 305, and an input/outputI/F 306, an audio input/output I/F 307, a network I/F 308, a short-rangecommunication circuit 309, an antenna 309 a for the short-rangecommunication circuit 309, an external device connection I/F 311, and abus line 310.

The CPU 301 controls overall operation of the robot 10. The CPU 301 isan arithmetic unit that reads programs or data from the ROM 302 or ahard disk (HD) 304 a onto the RAM 303, and executes processing accordingto the programs or data to implement functions of the robot 10.

The ROM 302 is a nonvolatile memory, which holds programs or data evenafter the robot 10 is turned off as the power is not supplied. The RAM303 is a volatile memory used as a work area for the CPU 301. The HDD304 controls reading or writing of various data from or to the HD 304 aunder control of the CPU 301. The HD 304 a stores various data such as acontrol program. The medium I/F 305 controls reading or writing(storage) of data to a storage medium 305 a such as a universal serialbus (USB) memory, a memory card, an optical disk, or a flash memory.

The input/output I/F 306 is an interface for controlling the output andinput of characters, numerical values, various instructions, and thelike to and from various external devices. The input/output I/F 306controls display of various information such as a cursor, a menu, awindow, characters, or an image on the display 150 such as a liquidcrystal display (LCD). In one example, the display 150 is a touch paneldisplay provided with an input device (input means). In another example,the input/output I/F 306 is connected to an input device (input means)such as a mouse and a keyboard, in addition to the display 150. Theaudio input/output I/F 307 is a circuit for controlling input and outputof audio signals between a microphone 307 a and the speaker 307 b undercontrol of the CPU 301. The microphone 307 a is an example of a soundcollecting device (sound collecting means), which is a built-in type,configured to input sound under control of the CPU 301. The speaker 308b is an example of a reproducing device (reproducing means) configuredto output an audio signal under control of the CPU 301.

The network I/F 308 is a communication interface that allows the robot10 to communicate (connect) with other devices or apparatuses throughthe communication network 9. The network I/F 308 is, for example, acommunication interface such as a wired or wireless LAN. In anotherexample, the network I/F 308 includes a communication interface such as3rd Generation (3G), Long Term Evolution (LTE), 4th Generation (4G), 5thGeneration (5G), Wi-Fi, Worldwide Interoperability for Microwave Access(WiMAX), Zigbee (registered trademark), or millimeter wave wirelesscommunication. The short-range communication circuit 309 is acommunication circuit that communicates in compliance with the nearfield communication (NFC) (Registered Trademark), the Bluetooth(Registered Trademark), and the like. The external device connection I/F311 is an interface circuit that connects the control device 30 toexternal devices.

Examples of the bus line 310 include, but not limited to, an address busand a data bus, which electrically connects the above-described hardwarecomponents. The bus line 310 transmits address signals, data signals,various control signals, and the like. The CPU 301, the ROM 302, the RAM303, the HDD 304, the medium I/F 305, the input/output I/F 306, theaudio input/output I/F 307, the network I/F 308, the short-rangecommunication circuit 309, and the external device connection I/F 311are connected to each other through the bus line 310.

Further, a drive motor 101, an actuator 102, an acceleration andorientation sensor 103, a global positioning system (GPS) receiver 104,the special image capturing device 21, a generic image capturing device24, a power supply unit 105, and the indicator lamps 160 are connectedto the control device 30 through the external device connection I/F 311.

The drive motor 101 drives the movement mechanism 17 to rotate inaccordance with an instruction from the CPU 301 to move the robot 10 onthe ground. The actuator 102 deforms the movable arm 11 in accordancewith an instruction from the CPU 301. Examples of the acceleration andorientation sensor 103 include, but not limited to, an electromagneticcompass or gyrocompass for detecting geomagnetism, and an accelerationsensor. The GPS receiver 104 receives a GPS signal from a GPS satellite.The power supply unit 105 supplies power to the entirety of the robot10. The control device 30 is an example of a communication deviceincluding the special image capturing device 21 and the generic imagecapturing device 24.

Hardware Configuration of Display Terminal:

FIG. 12 is a block diagram illustrating an example of a hardwareconfiguration of the display terminal 50, according to an embodiment.The display terminal 50 includes a CPU 501, a ROM 502, a RAM 503, anelectrically erasable programmable read-only memory (EEPROM) 504, animaging element I/F 505, a complementary metal oxide semiconductor(CMOS) 505 a, and a medium I/F 506.

The CPU 501 controls overall operation of the display terminal 50. TheCPU 501 is an arithmetic unit that reads programs or data from the ROM502, for example, onto the RAM 503, and executes processing according tothe programs or data to implement functions of the display terminal 50.

The ROM 502 stores programs such as an initial program loader (IPL) toboot the CPU 501. The RAM 503 is used as a work area for the CPU 501.The EEPROM 504 reads or writes various data such as a control programfor the display terminal 50 under control of the CPU 501.

The CMOS sensor 505 a captures a subject (mainly, the user operating thedisplay terminal 50) under control of the CPU 501 to obtain image data.The imaging element I/F 505 is a circuit that controls driving of theCMOS sensor 505 a. The medium I/F 506 controls reading or writing(storing) of data with respect to a storage medium 506 a such as a flashmemory.

Further, the display terminal 50 includes a network I/F 507, an audioinput/output I/F 508, a microphone 508 a, a speaker 508 b, a display511, a keyboard 512, an external device connection I/F 514, ashort-range communication circuit 515, and an antenna 515 a for theshort-range communication circuit 515.

The network I/F 507 is a communication interface that allows the displayterminal 50 to communicate (connect) with other devices or apparatusesthrough the communication network 9. The network I/F 507 is, forexample, a communication interface such as a wired or wireless LAN. Inanother example, the network I/F 507 includes a communication interfacesuch as 3G, LTE, 4G, 5G, Wi-Fi, WiMAX, Zigbee, or millimeter wavewireless communication. The audio input/output I/F 508 is a circuit forcontrolling input and output of audio signals between a microphone 508 aand the speaker 508 b under control of the CPU 501. The microphone 508 ais an example of a sound collecting device (sound collecting means),which is a built-in type, configured to input sound under control of theCPU 501. The speaker 508 b is an example of a reproducing device(reproducing means) configured to output an audio signal under controlof the CPU 501.

The display 511 is an example of a display unit configured to display animage of a subject, various kinds of icons, etc. Examples of the display511 include, but not limited to, an LCD and an organicelectroluminescence display. The keyboard 512 is one example of an inputdevice (input means) provided with a plurality of keys that allows auser to input characters, numerals, or various instructions. Theexternal device connection I/F 514 is an interface circuit that connectsthe display terminal 50 to various external devices. The external deviceconnection I/F 514 is connected to a mouse 500 that allows a user toinput an instruction such as selecting and executing various functions,selecting a target for processing, and moving a cursor. The mouse 500 isan example of an external input device (external input means), which isa pointing device that controls the display screen displayed on thedisplay 511. The mouse 500 is just one example of the pointing device.Other examples of the pointing device include a touch panel (touchscreen), a pen tablet, a touch pad, and a controller such as a joypad ora joystick. The short-range communication circuit 515 is a communicationcircuit that communicates in compliance with the NFC, the Bluetooth, andthe like.

The display terminal 50 further includes a bus line 509. The bus line509 is, for example, an address bus or a data bus, which electricallyconnects the components such as the CPU 501 illustrated in FIG. 12.

Hardware Configuration of Communication Management Server:

FIG. 13 is a block diagram illustrating an example of a hardwareconfiguration of the communication management server 90, according to anembodiment. The communication management server 90 is implemented by ageneral-purpose computer. The communication management server 90includes a CPU 901, a ROM 902, a RAM 903, an HD 904, an HDD 905, amedium I/F 907, a network I/F 908, a display 911, a keyboard 912, amouse 913, a digital versatile disk rewritable (DVD-RW) drive 915, and abus line 910.

The CPU 901 controls overall operation of the communication managementserver 90. The ROM 902 stores a control program for operating the CPU901. The RAM 903 is used as a work area for the CPU 901. The HDD 905controls reading or writing of various data to or from the HD 904 undercontrol of the CPU 901. The HD 904 stores various data such as a controlprogram. The medium I/F 907 controls reading or writing of data withrespect to a storage medium 906 such as a flash memory.

The network I/F 908 is an interface that controls communication of datathrough the communication network 9. The display 911 displays variousinformation such as a cursor, menu, window, characters, or image. Thekeyboard 912 is one example of an input device (input means) providedwith a plurality of keys that allows a user to input characters,numerals, or various instructions. The mouse 913 is one example of aninput device (input means) that allows a user to select a specificinstruction or execution, select a target for processing, or move acursor being displayed. The DVD-RW drive 915 controls reading or writingof various data from or to a DVD-RW 914, which is an example of aremovable storage medium. The removable storage medium is not limited tothe DVD-RW 914. In another example, a DVD-R can be used as the removalstorage medium. In still another example, in alternative to or inaddition to the DVD-RW drive 915, a Blu-ray (registered trademark) driveor a compact disc rewritable (CD-RW) drive are used to control readingor writing (storing) of data with respect to a Blu-ray disc rewritable(BD-RE) or a CD-RW.

The communication management server 90 further includes the bus line910. The bus line 910 is, for example, an address bus or a data bus,which electrically connects the components such as the CPU 901illustrated in FIG. 13.

Further, any one of the above-described control programs may be recordedin a file in a format installable or executable on a computer-readablestorage medium for distribution. Examples of the storage medium include,but not limited to, compact disc-recordable (CD-R), DVD, Blu-ray disc,and secure digital (SD) card. In addition, such storage medium may beprovided in the form of a program product to users within a certaincountry or outside that country. For example, the display terminal 50implements the control method according to the present disclosure byexecuting the program according to the present disclosure.

Hardware Configuration of Special Image Capturing Device:

A description is now given of a hardware configuration of the specialimage capturing device 21, with reference to FIG. 14. FIG. 14 is a blockdiagram illustrating an example of a hardware configuration of thespecial image capturing device 21, according to an embodiment. Thefollowing describes a case in which the special image capturing device21 is a spherical (omnidirectional) image capturing device having twoimaging elements. However, the special image capturing device 21 caninclude any suitable number of imaging elements, providing that itincludes at least two imaging elements. In addition, the special imagecapturing device 21 is not necessarily an image capturing devicededicated to omnidirectional image capturing. In another example, anexternal omnidirectional imaging unit is attached to a general-purposedigital camera or a smartphone to implement an image capturing devicehaving substantially the same function as that of the spherical imagecapturing device.

As illustrated in FIG. 14, the special image capturing device 21includes an imaging unit 201, an image processor 204, an imagingcontroller 205, a microphone 208, an audio processor 209, a CPU 211, aROM 212, and a static random access memory (SRAM) 213, a dynamic randomaccess memory (DRAM) 214, an operation unit 215, an input/output I/F216, a short-range communication circuit 217, an antenna 217 a for theshort-range communication circuit 217, and an acceleration andorientation sensor 218.

The imaging unit 201 includes two wide-angle lenses (so-called fisheyelenses) 202 a and 202 b, each having an angle of view of equal to orgreater than 180 degrees to form a hemispherical image. The imaging unit201 further includes two imaging elements 203 a and 203 b correspondingto the wide-angle lenses 202 a and 202 b respectively. Each of theimaging elements 203 a and 203 b includes an imaging sensor such as aCMOS sensor and a charge-coupled device (CCD) sensor, a timinggeneration circuit, and a group of registers. The imaging sensorconverts an optical image formed by the fisheye lenses 202 a and 202 binto electric signals to output image data. The timing generationcircuit generates horizontal or vertical synchronization signals, pixelclocks and the like for the imaging sensor. Various commands,parameters, and the like for operations of the imaging elements 203 aand 203 b are set in the group of registers.

Each of the imaging elements 203 a and 203 b of the imaging unit 201 isconnected to the image processor 204 via a parallel I/F bus. Inaddition, each of the imaging elements 203 a and 203 b of the imagingunit 201 is connected to the imaging controller 205 via a serial I/F bussuch as an I2C bus. The image processor 204, the imaging controller 205,and the audio processor 209 are each connected to the CPU 211 via a bus210. Further, the ROM 212, the SRAM 213, the DRAM 214, the operationunit 215, the input/output I/F 216, the short-range communicationcircuit 217, the acceleration and orientation sensor 218 are alsoconnected to the bus 210.

The image processor 204 acquires image data from each of the imagingelements 203 a and 203 b via the parallel I/F bus and performspredetermined processing on each image data. Thereafter, the imageprocessor 204 combines these image data to generate data of anequirectangular projection image.

The imaging controller 205 usually functions as a master device whileeach of the imaging elements 203 a and 203 b usually functions as aslave device. The imaging controller 205 sets commands or the like inthe group of registers of each of the imaging elements 203 a and 203 bvia the I2C bus. The imaging controller 205 receives various commandsfrom the CPU 211. Further, the imaging controller 205 obtains statusdata of the group of registers of each of the imaging elements 203 a and203 b via the I2C bus, and sends the obtained status data to the CPU211.

The imaging controller 205 instructs the imaging elements 203 a and 203b to output the image data at a time when the shutter button of theoperation unit 215 is pressed. In some cases, the special imagecapturing device 21 is configured to display a preview image on adisplay (e.g., a display of an external terminal such as a smartphonethat performs short-range communication with the special image capturingdevice 21 through the short-range communication circuit 217) or displaya moving image (movie). In case of displaying movie, image data arecontinuously output from the imaging elements 203 a and 203 b at apredetermined frame rate (frames per minute).

Furthermore, the imaging controller 205 functions as synchronizationcontrol means configured to operate in cooperation with the CPU 211, tosynchronize the time when the imaging element 203 a outputs image dataand the time when the imaging element 203 b outputs the image data.Although in the present embodiment, the special image capturing device21 does not include a display, the special image capturing device 21 mayinclude a display unit. The microphone 208 converts sound into audiodata (signal). The audio processor 209 obtains audio data output fromthe microphone 208 via an I/F bus and performs predetermined processingon the audio data.

The CPU 211 controls overall operation of the special image capturingdevice 21 and performs necessary processing. The ROM 212 stores variousprograms for the CPU 211. Each of the SRAM 213 and the DRAM 214 operatesas a work memory to store programs for execution by the CPU 211 or datain current processing. More specifically, in one example, the DRAM 214stores image data currently processed by the image processor 204 anddata of the equirectangular projection image on which processing hasbeen performed.

The operation unit 215 collectively refers to various operation keys,such as a shutter button. The user operates the operation unit 215 toinput various image capturing (photographing) modes or image capturing(photographing) conditions. The input/output I/F 216 collectively refersto an interface circuit such as a USB I/F with an external medium suchas an SD card or an external personal computer. The input/output I/F 216may be either wired or wireless. The data of the equirectangularprojection image, which is stored in the DRAM 214, is stored in theexternal medium via the input/output I/F 216 or transmitted to anexternal terminal (apparatus) via the input/output I/F 216, as needed.The short-range communication circuit 217 communicates with an externalterminal (apparatus) via the antenna 217 a of the special imagecapturing device 21 through a short-range wireless communication networksuch as Wi-Fi, NFC, and Bluetooth (registered trademark). Theshort-range communication circuit 217 is configured to transmit the dataof equirectangular projection image to an external device (apparatus).

The acceleration and orientation sensor 218 calculates an orientation ofthe special image capturing device 21 from the Earth's magnetism tooutput orientation and tilt information. This orientation and tiltinformation is an example of related information, which is metadatadescribed in compliance with Exif. This information is used for imageprocessing such as image correction of captured images. The relatedinformation also includes data of a time (date) when an image iscaptured by the special image capturing device 21, and data size ofimage data, for example. The acceleration and orientation sensor 218detects the change in tilt (roll, pitch, yaw) with movement of thespecial image capturing device 21. The change in tilt is one example ofrelated information (metadata) described in compliance with Exif. Thisinformation is used for image processing such as image correction ofcaptured images. The acceleration and orientation sensor 218 furtherdetects acceleration in three axial directions. The special imagecapturing device 21 calculates the position (an angle with respect tothe direction of gravity) of the special image capturing device 21,based on the acceleration detected by the acceleration and orientationsensor 218. With the acceleration and orientation sensor 218, thespecial image capturing device 21 is able to correct images with highaccuracy.

Spherical Image:

A description is now given of an example of a spherical image obtainedby the special image capturing device 21, with reference to FIG. 15A toFIG. 15C to FIG. 19. First, referring to FIG. 15A to FIG. 15C, FIG. 16A,and FIG. 16B, a description is given of an overview of an operation ofgenerating an equirectangular projection image EC and a spherical imageCE from the images captured by the special image capturing device 21.FIG. 15A is an illustration of a hemispherical image (front side)captured by the special image capturing device 21. FIG. 15B is anillustration of a hemispherical image (back side) captured by thespecial image capturing device 21. FIG. 15C is an illustration of animage in equirectangular projection, which is referred to as an“equirectangular projection image” (or equidistant cylindricalprojection image) EC. FIG. 16A is a conceptual diagram illustrating anexample of how the equirectangular projection image maps to a surface ofa sphere. FIG. 16B is an illustration of the spherical image.

As illustrated in FIG. 15A, an image captured by the imaging element 203a is a curved hemispherical image (front side) taken through the fisheyelens 202 a. Also, as illustrated in FIG. 15B, an image captured by theimaging element 203 b is a curved hemispherical image (back side) takenthrough the fisheye lens 202 b. The special image capturing device 21combines the hemispherical image (front side) and the hemisphericalimage (rear side), which is 180 degrees reversed, to generate anequirectangular projection image as illustrated in FIG. 15C.

The special image capturing device 21 uses Open Graphics Library forEmbedded Systems (OpenGL ES) to map the equirectangular projection imageon the sphere surface as illustrated in FIG. 16A, whereby generating thespherical image CE as illustrated in FIG. 16B. Thus, the spherical imageCE is represented as the equirectangular projection image EC, whichcorresponds to a surface facing a center of the sphere CS. It should benoted that OpenGL ES is a graphic library used for visualizingtwo-dimensional (2D) and three-dimensional (3D) data. The sphericalimage CE is either a still image or a moving image (movie).

Since the spherical image CE is an image attached to the sphere surface,a part of the image may look distorted when viewed from the user,providing a feeling of strangeness. To resolve this strange feeling, thespecial image capturing device 21 controls a particular display todisplay an image of a viewable area, which is a part of the sphericalimage CE, as a flat image having fewer curves. The viewable area is, forexample, a part of the spherical image CE that is viewable by the user.In this disclosure, the image of the viewable area is referred to as a“viewable-area image”. A description is now given of displaying theviewable-area image, with reference to FIG. 17, FIG. 18A, and FIG. 18B.

FIG. 17 is an illustration of relative positions of a virtual camera ICand a viewable area in a case where the spherical image CE isrepresented as a surface area of a three-dimensional solid sphere CS.The virtual camera IC corresponds to a position of a point of view(viewpoint) of a user who is viewing the spherical image CE representedas a surface area of the three-dimensional solid sphere. FIG. 18A is aperspective view of FIG. 17. FIG. 18B is a view illustrating theviewable-area image when displayed on a display. FIG. 18A illustratesthe spherical image CE illustrated in FIG. 17 as a three-dimensionalsolid sphere CS. Assuming that the spherical image CE is a surface areaof the solid sphere CS, the virtual camera IC is inside of the sphericalimage CE as illustrated in FIG. 17. The viewable area T in the sphericalimage CE is an imaging area of the virtual camera IC. Specifically, theviewable area T is specified by viewable-area information indicating animaging direction and an angle of view of the virtual camera IC in athree-dimensional virtual space containing the spherical image CE. Inaddition, zooming in the viewable area T can also be determined bybringing the virtual camera IC closer to or away from the sphericalimage CE. A viewable-area image Q is an image of the viewable area T inthe spherical image CE. The viewable area T is specified by the angle ofview α and a distance f from the virtual camera IC to the sphericalimage CE (see FIG. 19).

The viewable-area image Q illustrated in FIG. 18A, is displayed on adisplay as an image of an imaging area of the virtual camera IC, asillustrated in FIG. 18B. FIG. 18B illustrates the viewable-area image Qrepresented by the viewable-area information that is set by default. Inthe following, the position of the virtual camera IC is described usingan imaging direction (ea, aa) and an angle of view α of the virtualcamera IC. In another example, the viewable area T is identified by animaging area (X, Y, Z) of the virtual camera IC, i.e., the viewable areaT, rather than the angle of view α and the distance f.

Referring to FIG. 19, a relation between the viewable-area informationand the image of the viewable area T is described. FIG. 19 is a viewillustrating a relation between the viewable-area information and theimage of the viewable area T. As illustrated in FIG. 19, “ea” denotes anelevation angle, “aa” denotes an azimuth angle, and “α” denotes an angleof view. The position of the virtual camera IC is adjusted, such thatthe point of gaze of the virtual camera IC, indicated by the imagingdirection (ea, aa), matches the center point CP of the viewable area Tas the imaging area of the virtual camera IC. As illustrated in FIG. 19,when it is assumed that a diagonal angle of the viewable area Tspecified by the angle of view α of the virtual camera IC is α, thecenter point CP provides the parameters (x, y) of the viewable-areainformation. The viewable-area image Q is an image of the viewable areaT in the spherical image CE. “f” denotes a distance from the virtualcamera IC to the center point CP. “L” denotes a distance between thecenter point CP and a given vertex of the viewable area T (2L is adiagonal line). In FIG. 19, a trigonometric function equation generallyexpressed by the following equation 1 is satisfied.

L/f=tan(α/2)  (Equation 1)

Functional Configuration:

A description is now given of a functional configuration of the remotecontrol system 1 a, according to the present embodiment, with referenceto FIG. 20 to FIG. 25A and FIG. 25B. FIG. 20 is a block diagramillustrating an example of a functional configuration of the remotecontrol system 1 a, according to an embodiment. FIG. 20 illustrates aterminal, an apparatus, and a server that relate to processes oroperations to be described below among the terminals, apparatuses, andservers illustrated in FIG. 1.

Functional Configuration of Control Device:

First, referring to FIG. 20, the functional configuration of the controldevice 30 that controls processing or operation of the robot 10 isdescribed. The control device 30 includes a data exchange unit 31, anacceptance unit 32, a display control unit 33, a determination unit 34,a state information generation unit 35, a position information detectionunit 36, an image capturing control unit 37, a captured imageacquisition unit 38, a movement control unit 41, an arm operationcontrol unit 42, an audio input/output unit 43, and a storage/readingunit 39. These units are functions or means implemented by or that arecaused to function by operating any of the hardware elements illustratedin FIG. 11 in cooperation with the instructions of the CPU 301 accordingto the control program expanded to the RAM 303. The control device 30further includes a storage unit 3000, which is implemented by the ROM302, the HD 304 a or the storage medium 305 a illustrated in FIG. 11.

The data exchange unit 31, which is implemented by instructions of theCPU 301 and by the network I/F 308 illustrated in FIG. 11, has afunction of transmitting and receiving various types of data orinformation to and from other apparatuses through the communicationnetwork 9. The data exchange unit 31 transmits spherical image data ordetailed image data acquired by the captured image acquisition unit 38to the display terminal 50 via the communication network 9, for example.

The acceptance unit 32, which is implemented by instructions of the CPU301 and the input/output I/F 306 illustrated in FIG. 11, has a functionof receiving an operation input to an input device (input means) such asthe display 150. The display control unit 33, which is implemented byinstructions of the CPU 301 and input/output I/F 306 illustrated in FIG.11, has a function of controlling the display 150 to display variousscreen. The determination unit 34, which is implemented by instructionsof the CPU 301 illustrated in FIG. 11, has a function of making variousdeterminations.

The state information generation unit 35, which is implemented byinstructions of the CPU 301 and the external device connection I/F 311illustrated in FIG. 11, has a function of generating state information170 that indicates the movement state of the robot 10. The stateinformation generation unit 35 generates and acquires the stateinformation 170 indicating the movement state of the robot 10 based onthe driving state of the movement mechanism 17 acquired from themovement control unit 41. A detailed description is given below of thestate information 170 generated (acquired) by the state informationgeneration unit 35.

The position information detection unit 36, which is implemented byinstructions of the CPU 301 and the external device connection I/F 311illustrated in FIG. 11, has a function of acquiring a detection resultof directions of bearings (e.g., azimuth angle, magnetic north) detectedby the acceleration and orientation sensor 103 or the GPS receiver 104.The detection result of the directions of bearings is positioninformation indicating the position and the orientation of the robot 10at a particular time.

The image capturing control unit 37, which is implemented byinstructions of the CPU 301 and the external device connection I/F 311illustrated in FIG. 11, has a function of instructing the special imagecapturing device 21 or the generic image capturing device 24 to performimage capturing processing. For example, the image capturing controlunit 37 transmits instruction information for instructing imagecapturing by the special image capturing device 21 to the special imagecapturing device 21. Further, for example, the image capturing controlunit 37 transmits instruction information for instructing imagecapturing by the generic image capturing device 24 to the generic imagecapturing device 24.

The captured image acquisition unit 38, which is implemented byinstructions of the CPU 301 and the external device connection I/F 311illustrated in FIG. 11, has a function of acquiring a spherical imageacquired the special image capturing device 21 or a detailed imageacquired by the generic image capturing device 24. For example, thecaptured image acquisition unit 38 acquires, from the special imagecapturing device 21, spherical image data, which is data of a sphericalimage obtained by photographing a subject by the special image capturingdevice 21. Further, for example, the captured image acquisition unit 38acquires, from the generic image capturing device 24, detailed imagedata, which is data of a detailed image obtained by the generic imagecapturing device 24 by photographing a part of the subject photographedby the special image capturing device 21.

The movement control unit 41, which is implemented by instructions ofthe CPU 301 and the external device connection I/F 311 illustrated inFIG. 11, has a function of drive the movement mechanism 17 to controlthe movement of the robot 10. For example, the movement control unit 41controls the driving of the movement mechanism 17 in response to arequest command transmitted from the display terminal 50, to move therobot 10.

The arm operation control unit 42, which is implemented by instructionsof the CPU 301 and the external device connection I/F 311 illustrated inFIG. 11, has a function of controlling operation of the movable arm 11.For example, the arm operation control unit 42 deforms the movable arm11 in response to a request command transmitted from the displayterminal 50, to change the direction or position of the movable arm 11.

The audio input/output unit 43, which is implemented by instructions ofthe CPU 301 and the audio input/output I/F 307 illustrated in FIG. 11,has a function of inputting and outputting audio data. For example, theaudio input/output unit 43 converts a user's voice collected by themicrophone 307 a into an audio signal, and performs audio processing onaudio data based on the audio signal. Further, for example, the audioinput/output unit 43 sets audio data to be output from the speaker 307b, such that the speaker 307 b outputs the audio signal based on the setaudio data to reproduce sound.

The storage/reading unit 39, which is implemented by instructions of theCPU 301 illustrated in FIG. 11, has a function of storing various datain the storage unit 3000, and reads various data from the storage unit3000. Further, every time image data and audio data are received inperforming communication with the display terminal 50, thestorage/reading unit 39 overwrites data stored in the storage unit 3000with the received image data and audio data. The display 150 displays animage based on image data before being overwritten, and the speaker 307b outputs sound based on audio data before being overwritten. Thestorage unit 3000 further stores spherical image data and detailed imagedata acquired by the captured image acquisition unit 38. In one example,a spherical image data item and a detailed image data item stored in thestorage unit 3000 are deleted when a predetermined time period haselapsed since the data items are acquired by the captured imageacquisition unit 38. In another example, a spherical image data item anda detailed image data item stored in the storage unit 3000 are deletedafter the data items are transmitted to the display terminal 50.

Command Management Table:

FIG. 21A is a conceptual diagram illustrating an example of a commandmanagement table, according to an embodiment. In the storage unit 3000,a command management database (DB) 3001 storing the command managementtable as illustrated in FIG. 21A is stored. The command management tableillustrated in FIG. 21A is used for identifying processing or operationto be executed by the robot 10 based on a request command transmittedfrom the display terminal 50. The command management table stores, foreach of plural commands, a variable and a processing contentcorresponding to the command in association with each other. Thedetermination unit 34 of the control device 30 identifies particularprocessing corresponding to the request command transmitted from thedisplay terminal 50 using the command management table.

For example, the processing corresponding to a command “MOVE (variablesL, R)” is processing of rotating the left wheel of the movementmechanism 17 by L degrees and rotating the right wheel by R degrees.Although in embodiment, a description is given of an example in whichthe robot 10 moves with two separate left and right wheels, theembodiments are not limited thereto. In another example in which themovement mechanism 17 of a foot type or a single wheel is used, the sameprocessing is performed provided that it moves the robot 10 in aparticular direction. Further, the processing corresponding to thecommand “AIM (variables H, V, Z)” is processing of moving the imagingdirection of the generic image capturing device 24 so that the imagingdirection is the horizontal angle H degrees and the vertical angle Vdegrees with respect to the front of the robot 10 and changing the zoomamount of the generic image capturing device 24 to Z %. Further, thecommand “LOOK (variables H, V, Z)” indicates the direction (H, V) in thespherical image viewed by the operator of the robot 10 and the zoomamount (Z %). The robot 10 deals with this command “LOOK (variables H,V, Z)” as information transmitted from the display terminal 50, and, forexample, uses this command to generate a display screen 800 (see FIG.43A) described below to be displayed on the display 150.

Image Capturing Parameter Management Table:

FIG. 21B is a conceptual diagram illustrating an example of an imagecapturing parameter management table, according to an embodiment. In thestorage unit 3000, an image capturing parameter management DB 3002storing the image capturing parameter management table as illustrated inFIG. 21B is stored. The image capturing parameter management tableillustrated in FIG. 21B stores image capturing parameters such as theimage quality of the spherical image captured by the special imagecapturing device 21. The image capturing parameter management tablestores and manages, for each of the plural items, a parameter of theitem. The items stored in the image capturing parameter management tableinclude the frame rate (FPS, i.e., an update frequency per second) ofthe spherical image, the resolution of the spherical image, and theimaging position (AIM) of the generic image capturing device 24. Notethat the items stored in the image capturing parameter management tableare not limited to the above-described items. For example, other itemsregarding the image quality of the spherical image are included. Theimage capturing control unit 37 of the control device 30 updates(changes) each parameter each time the image quality of the sphericalimage acquired by the special image capturing device 21 is changedand/or each time the imaging position of the detailed image acquired bythe generic image capturing device 24 is changed.

Functional Configuration of Display Terminal:

Referring again to FIG. 20, a description is now given of the functionalconfiguration of the display terminal 50. The display terminal 50includes a data exchange unit 51, an acceptance unit 52, a displayscreen generation unit 53, a display control unit 54, a display positionidentification unit 55, a determination unit 56, a request commandgeneration unit 57, a captured image acquisition unit 58, an audioinput/output unit 61 and a storage/reading unit 59. These units arefunctions or means implemented by or that are caused to function byoperating any of the hardware elements illustrated in FIG. 12 incooperation with the instructions of the CPU 501 according to thecontrol program expanded to the RAM 503. The display terminal 50 furtherincludes a storage unit 5000, which is implemented by the ROM 502 or thestorage medium 506 a illustrated in FIG. 12. Further, in the displayterminal 50, a dedicated application program for remotely operating therobot 10 is installed. The display terminal 50 implements the functionsby causing the CPU 501 to execute the installed application program, forexample.

The data exchange unit 51, which is implemented by instructions of theCPU 501 and by the network I/F 507 illustrated in FIG. 12, has afunction of transmitting and receiving various types of data orinformation to and from other apparatuses through the communicationnetwork 9. For example, the data exchange unit 51 receives sphericalimage data and detailed image data from the robot 10 (control device 30)through the communication network 9. Further, for example, the dataexchange unit 51 receives (acquires) the state information 170indicating the state of the robot 10 from the robot 10 (control device30) through the communication network 9. Furthermore, for example, thedata exchange unit 51 transmits the request command generated by therequest command generation unit 57 to the robot 10 (control device 30)through the communication network 9.

The acceptance unit 52, which is implemented by instructions of the CPU501, and an input device (input means) such as the keyboard 512 or theexternal device connection I/F 514 illustrated in FIG. 12, has afunction of receiving various selections or operation inputs to thedisplay terminal 50. For example, the acceptance unit 52 accepts anoperation input on a display screen displayed on the display 511 byusing the input device such as the keyboard 512. Further, for example,the acceptance unit 52 detects movement of a pointing device such as themouse 500.

The display screen generation unit 53, which is implemented byinstructions of the CPU 501 illustrated in FIG. 12, generates variousdisplay screens to be displayed on the display 511. For example, thedisplay screen generation unit 53 generates a display screen todisplayed on the display 511 using spherical image data and detailedimage data transmitted from the robot 10.

The display control unit 54, which is implemented by instructions of theCPU 501 illustrated in FIG. 12, has a function of controlling thedisplay 511 of the display terminal 50 to display various screen. Forexample, the display control unit 54 controls the display 511 to displaya display screen generated by the display screen generation unit 53, tooutput the display screen.

The display position identification unit 55, which is implemented byinstructions of the CPU 501 and the external device connection I/F 514illustrated in FIG. 12, has a function of identifying a display position(display direction) of the spherical image displayed on the display 511.For example, the display position identification unit 55 calculates, forexample, an amount of movement of the mouse 500 accepted by theacceptance unit 52. Further, for example, the display positionidentification unit 55 identifies the display position (displaydirection) of the spherical image based on the amount of movement of themouse 500.

The determination unit 56, which is implemented by instructions of theCPU 501 illustrated in FIG. 12, has a function of making variousdeterminations. The determination unit 56 determines a particularprocessing to be requested to the robot 10. For example, thedetermination unit 56 determines the particular processing to berequested to the robot 10 based on an operation input accepted by theacceptance unit 52. Further, for example, the determination unit 56determines the particular processing to be requested to the robot 10based on the state information 170 received (acquired) by the dataexchange unit 51.

The request command generation unit 57, which is implemented byinstructions of the CPU 501 illustrated in FIG. 12, has a function ofgenerating a request command, which is an execution request for causingthe robot 10 to execute a particular processing. For example, therequest command generation unit 57 generates a request command that is arequest for changing the imaging position of the generic image capturingdevice 24 of the robot 10.

The captured image acquisition unit 58, which is implemented byinstructions of the CPU 501 and the imaging element I/F 505 illustratedin FIG. 12, has a function of acquiring a captured image captured by theCMOS sensor 505 a. For example, the captured image acquisition unit 58acquires captured-image data, which is data of a captured image acquiredby capturing an image of a subject by the CMOS sensor 505 a, forexample. The audio input/output unit 61, which is implemented byinstructions of the CPU 501 and the audio input/output I/F 508illustrated in FIG. 12, has a function of inputting and outputting audiodata. For example, the audio input/output unit 61 converts a user'svoice collected by the microphone 508 a into an audio signal, andperforms audio processing on audio data based on the audio signal.Further, for example, the audio input/output unit 61 sets audio data tobe output from the speaker 508 b, such that the speaker 508 b outputsthe audio signal based on the set audio data to reproduce sound.

The storage/reading unit 59, which is implemented by instructions of theCPU 501 illustrated in FIG. 12, has a function of storing various datain the storage unit 5000, and reads various data from the storage unit5000. Further, every time image data and audio data are received inperforming communication with robot 10 (the control device 30), thestorage/reading unit 59 overwrites data stored in the storage unit 5000with the received image data and audio data. The display 511 displays animage based on image data before being overwritten, and the speaker 508b outputs sound based on audio data before being overwritten.

State Information Management Table:

FIG. 22A is a conceptual diagram illustrating an example of a stateinformation management table, according to an embodiment. In the storageunit 5000, a state information management DB 5001 storing the stateinformation management table as illustrated in FIG. 22A is stored. Thestate information management table illustrated in FIG. 22A stores thecurrent state of the robot 10. This state information management tablestores, for each of a drive direction of the robot 10 and a drive speedof the robot 10, values (parameters) indicating the current movementstate of the robot 10. The drive direction of the robot 10 is indicatedby a horizontal angle (DRIVE_H_ANGLE) and a vertical angle(DRIVE_V_ANGLE). The state information management table further storesvalues (parameters) indicating the current imaging position of thegeneric image capturing device 24. The imaging position of the genericimage capturing device 24 is indicated by a horizontal angle(CAMERA_H_ANGLE), a vertical angle (CAMERA_V_ANGLE), and a zoom amount(CAMERA_ZOOM). Each time the robot 10 moves or each time the imagingposition of the generic image capturing device 24 is changed, thedisplay terminal 50 updates (changes) the value of the correspondingitem.

Condition Information Management Table:

FIG. 22B is a conceptual diagram illustrating an example of a conditioninformation management table, according to an embodiment. In the storageunit 5000, a condition information management DB 5002 storing thecondition information management table as illustrated in FIG. 22B isstored. The condition information management table illustrated in FIG.22B is used to identify a content of processing or operation to berequested to the robot 10 based on the state information 170 received bythe data exchange unit 51. The condition information management tablestores, for each of conditions relating to the state of the robot 10, acontent of processing in association with the request command to betransmitted to the robot 10. For example, when the drive speed of therobot 10 is equal to or more than 5.0 km/h (“SPEED>5 km/h”), the displayterminal 50 identifies a processing of “reducing the frame rate to 3” asthe content of the processing, and extracts “FPS(3)” as the requestcommand to be transmitted to the robot 10. “RESOLUTION” is a commandrelating to a resolution of the spherical image. “ZOOM” is a commandrelating to a display range (output range) of the spherical image.

The conditions indicated in the condition information management tableare not limited to the conditions relating to the drive speed of therobot 10, and, in one example, a condition regarding other informationof the moving state acquired by the robot 10 is indicated. For example,the condition information management table indicates a conditionsrelating to the drive direction of the robot 10. Further, an operatorwho operates the robot 10 using the display terminal 50 is allowed tomodify or change the conditions and/or the contents of the processingindicated in the condition information management table as appropriate.

Further, as indicated in the condition information management table,when the display position of the spherical image and the imagingposition of the generic image capturing device 24 are close to eachother, the display terminal 50 switches the display screen from thespherical image to the detailed image. This means that when by changingthe imaging position of the generic image capturing device 24, theimaging direction and zoom amount of the generic image capturing device24 catches up to the display position (display direction and zoomamount) of the spherical image, the display on the display 511 isswitched from the spherical image to the detailed image. In theembodiment, “the display position of the spherical image and the imagingposition of the generic image capturing device 24 are close to eachother” is not limited to that the positions are completely the same. Itsuffices that the positions are within a predetermined range. Thispredetermined range is set by a designer or a user, for example. Inanother example, the predetermined range is determined according to aparameter such as the degree of matching or the degree of similaritybetween the displayed spherical image and the detailed imagephotographed according to the imaging position of the generic imagecapturing device 24. In this case, for example, when the parameter suchas the degree of matching or the degree of similarity between thespherical image and the detailed image is equal to or more than a setvalue, the display terminal 50 determines that the imaging position ofthe generic image capturing device 24 is close to the display positionof the spherical image.

User Command Management Table:

FIG. 23A is a conceptual diagram illustrating an example of a usercommand management table, according to an embodiment. In the storageunit 5000, a user command management DB 5003 storing the user commandmanagement table as illustrated in FIG. 23A is stored. The user commandmanagement table illustrated in FIG. 23A is used to identify a contentof processing or operation to be requested to the robot 10 based on anoperation input accepted by the acceptance unit 52. The user commandmanagement table stores, for each of input commands, a content ofprocessing in association with a type of the processing. For example, inresponse to detecting an input command “movement of the mouse 500”, thedisplay terminal 50 transmits a request command of LOOK (H, V, Z) to therobot 10 and changes the display position of the spherical image.Further, for example, in response to detecting an input command “themouse 500 stops after moving”, the display terminal 50 transmits arequest command of AIM (H, V, Z) to the robot 10. This means that if aninstruction is given to the generic image capturing device 24 every timethe mouse 500 moves, the instructions are given too frequently, andtherefore the request command is transmitted when the mouse 500 is notmoved for a predetermined time period after the mouse 500 was movedlast.

Display State Management Table:

FIG. 23B is a conceptual diagram illustrating an example of a displaystate management table, according to an embodiment. In the storage unit5000, a display state management DB 5004 storing the display statemanagement table as illustrated in FIG. 23B is stored. The display statemanagement table illustrated in FIG. 23B stores and manages parametersof a display position of the spherical image displayed on the displayterminal 50 and the imaging position of the detailed image displayed onthe display terminal 50. For example, the table of FIG. 23B indicatesthat the display direction of the spherical image is the horizontalangle “22.0°” and the vertical angle “15.0°”, and the zoom amount of thespherical image is 50%. Further, the imaging direction of the detailedimage (the generic image capturing device 24) are horizontal angle“−22.2°”, and vertical angle “10.0°”, and the zoom amount of thedetailed image (the generic image capturing device 24) is 22.3%. Thedisplay terminal 50 switches an image to be displayed on the display 511by using the parameters of the display state parameter of each imagestored in the display state management table.

Functional Configuration of Communication Management Server:

Referring again to FIG. 20, a description is now given of the functionalconfiguration of the communication management server 90. Thecommunication management server 90 includes a data exchange unit 91, anauthentication unit 92, a determination unit 93, a generation unit 94,and a storage/reading unit 99. These units are functions or meansimplemented by or that are caused to function by operating any of thehardware elements illustrated in FIG. 13 in cooperation with theinstructions of the CPU 901 according to the control program expanded tothe RAM 903. The communication management server 90 further includes astorage unit 9000, which is implemented by the ROM 902, the HD 904, orthe storage medium 906 illustrated in FIG. 13.

The data exchange unit 91, which is implemented by instructions of theCPU 901 and by the network I/F 908 illustrated in FIG. 13, has afunction of transmitting and receiving various types of data orinformation to and from other apparatuses through the communicationnetwork 9.

The authentication unit 92, which is implemented by instructions of theCPU 901 illustrated in FIG. 13, has a function of authenticating a loginrequest source based on login request information received by the dataexchange unit 91. For example, the authentication unit 92 searches anauthentication management DB 9001 in the storage unit 9000 using aterminal identification (ID) and a password included in the loginrequest information received by the data exchange unit 91 as searchkeys. The authentication unit 92 determines whether the authenticationmanagement DB 9001 stores a combination of a terminal ID and a passwordidentical to the terminal ID and the password included in the loginrequest information, to perform a terminal authentication.

The determination unit 93, which is implemented by instructions of theCPU 901 illustrated in FIG. 13, has a function of determining whether aterminal ID of the display terminal 50 is stored in a session managementtable described below. The generation unit 94, which is implemented byinstructions of the CPU 901 illustrated in FIG. 13, generates a sessionID used for communication.

The storage/reading unit 99, which is implemented by instructions of theCPU 901 illustrated in FIG. 13, has a function of storing various datain the storage unit 9000, and reads various data from the storage unit9000. Further, the storage unit 9000 includes destination list framedata (icons and destination list content information such as “rA01”,“robot 10A-1” illustrated in FIG. 27 are not included) in a destinationlist screen 900 (see FIG. 27) described below.

Authentication Management Table:

FIG. 24A is a conceptual diagram illustrating an example of anauthentication management table, according to an embodiment. In thestorage unit 9000, an authentication management DB 9001 storing theauthentication management table as illustrated in FIG. 24A is stored.The authentication management table stores, for each of terminal IDs ofall the display terminals 50 managed by the communication managementserver 90, the terminal ID in association with a password. For example,the authentication management table illustrated in FIG. 24A indicatesthat the terminal ID of the display terminal 50A is “o01” and itspassword is “aaaa”

Terminal Management Table:

FIG. 24B is a conceptual diagram illustrating an example of a terminalmanagement table, according to an embodiment. In the storage unit 9000,a terminal management DB 9002 storing the terminal management table asillustrated in FIG. 24B is stored. The terminal management table stores,for each of the terminal IDs of the terminals (the robots 10 and thedisplay terminals 50), a terminal name of the terminal, an IP address ofthe terminal, operating state information indicating a current operatingstate of the terminal, and a site name indicating a site where the robot10 is located in a case where the terminal is the robot 10, inassociation with one another. For example, the terminal management tableillustrated in FIG. 24B indicates that the display terminal 50 whoseterminal ID is “o01” has the terminal name “display terminal 50A”, theIP address of the display terminal 50 is “1.2.1.3”, and the operatingstate is “Online (Ready)”. Further, this terminal management tableindicates that the robot 10 whose terminal ID is “rA01” has the terminalname “robot 10A-1”, the IP address of the robot 10 is “1.3.2.3”, theoperating state is “Online (Ready), and the site name is “Site A”.

Destination List Management Table:

FIG. 25A is a conceptual diagram illustrating an example of adestination list management table, according to an embodiment. In thestorage unit 9000, a destination list management DB 9003 storing thedestination list management table as illustrated in FIG. 25A is stored.The destination list management table stores, for each of the displayterminals 50 as source terminals each requesting the start ofcommunication with the robot 10, the terminal ID of the display terminal50 in association with the terminal ID of the destination candidaterobots 10 registered as candidates of robots 10 as destination terminalof the communication. For example, the destination list management tableillustrated in FIG. 25A indicates that destination terminal candidatesto which the source terminal (display terminal 50A) whose terminal ID is“o01a” can send a communication start request are the robot 10A-1 whoseterminal ID is “rA01”, the robot 10A-2 whose terminal ID is “rA02”, andthe robot 10C-1 whose terminal ID is “rC01”. The terminal IDs of therobots 10 as the destination terminal candidates are updated by additionor deletion in response to an addition or deletion request transmittedfrom any source terminal (display terminal 50) to the communicationmanagement server 90.

Session Management Table:

FIG. 25B is a conceptual diagram illustrating an example of a sessionmanagement table, according to an embodiment. In the storage unit 9000,a session management DB 9004 storing the session management table asillustrated in FIG. 25B is stored. The session management table stores,for each of session IDs each identifying a session used when the robot10 and the display terminal 50 communicate with each other, the sessionID in association with the terminal IDs of the robot(s) 10 and thedisplay terminal(s) 50 using the session identified by the associatedsession ID. For example, in the session management table illustrated inFIG. 25B indicates that terminals using the session executed using thesession ID “se1” are the display terminal 50A whose terminal ID is“o01”, the robot 10A-2 whose terminal ID is “rA02”, and the robot 10C-1whose terminal ID is “rC01”.

Processes or Operation of Embodiment:

A description is now given of processes or operation performed by theremote control system 1 a, according to an embodiment, with reference toFIG. 26 to FIG. 43A and FIG. 43B. In the following description,processes performed by the control device 30 of the robot 10 aredescribed as processes performed by the robot 10.

Establishing Communication Session:

First, a description is given of operation of establishing acommunication session between the robot 10 and the display terminal 50,with reference to FIG. 26 to FIG. 28. FIG. 26 is a sequence diagramillustrating an example of operation performed at preparatory stage forstarting data exchange between the robot 10 and the display terminal 50in the remote control system 1 a, according to an embodiment. In theembodiment, a description is given of operation of exchanging managementinformation items at the preparation stage performed before startingdata exchange between the display terminal 50A as a source terminal andthe robot 10A-1 as a destination terminal.

First, the data exchange unit 51 of the display terminal 50A transmitslogin request information indicating a login authentication request tothe communication management server 90 through the communication network9 (step S101). Specifically, when the user of the display terminal 50Aturns on the power switch of the display terminal 50A, the power isturned on. In response the power being turned on, the data exchange unit51 of the display terminal 50A transmits the login request informationfrom the data exchange unit 51 to the communication management server 90through the communication network 9. Thereby, the data exchange unit 91of the communication management server 90 receives the login requestinformation transmitted from the display terminal 50A.

The login request information includes a terminal ID identifying thesource terminal as the display terminal 50A, and a password. Theterminal ID and the password are data that have been read by thestorage/reading unit 59 from the storage unit 5000 and sent to the dataexchange unit 51. The terminal ID and password are sent in any othersuitable manner than the above. In another example, a terminal ID and/ora password input by the user using the an input device (input means)such as the keyboard 512 is transmitted. In still another example, aterminal ID and/or a password read from a storage medium such as asubscriber identity module (SIM) card or an SD card connected to thedisplay terminal 50A is transmitted.

When the login request information is transmitted from the displayterminal 50A to the communication management server 90, thecommunication management server 90 on the receiving side acquires the IPaddress of the display terminal 50A on the transmitting side. The startof the login request does not necessarily have to be triggered byturning on the power switch. In another example, the login request istransmitted in response to an input to an input device (input means)such as the display 511.

Next, the authentication unit 92 of the communication management server90 searches the authentication management table (see FIG. 24A) stored inthe storage unit 9000 using the terminal ID and the password included inthe login request information received by the data exchange unit 91 assearch keys, to perform authentication by determining whether the sameterminal ID and the same password are stored in the authenticationmanagement DB 9001 (step S102). In the following, a description is givenof a case where the authentication unit 92 determines that the displayterminal 50A is a terminal that has a valid usage authorization.

When the authentication unit 92 determines that the same terminal ID andthe same password are stored and therefore the login request istransmitted from a source terminal having a valid usage authorization,the storage/reading unit 99 reads out the destination list frame datafrom the storage unit 9000 (step S103).

The data exchange unit 91 transmits an authentication result informationindicating an authentication result obtained by the authentication unit92 to the display terminal 50A, which is the login request sender,through the communication network 9 (step S104). The authenticationresult information includes the destination list frame data that is readout in step S103. Thereby, the data exchange unit 51 of the displayterminal 50A receives the authentication result information. Then, thestorage/reading unit 59 of the display terminal 50A stores thedestination list frame data received in step S104 in the storage unit5000 (step S105).

Next, in response to receiving the authentication result informationindicating the authentication result obtained by determination that theterminal has a valid usage authorization, the data exchange unit 51transmits destination list content request information requesting acontent of a destination list to the communication management server 90via the communication network 9 (step S106) This destination listcontent request information includes the terminal ID of the displayterminal 50A. Thereby, the data exchange unit 91 of the communicationmanagement server 90 receives the destination list content requestinformation.

Next, the storage/reading unit 99 of the communication management server90 searches the destination list management DB 9003 (FIG. 25A) using theterminal ID “o01” of the display terminal 50 received in step S106 as asearch key, to read terminal IDs of all the destination candidatesassociated with the terminal ID (step S107). Further, thestorage/reading unit 99 searches the terminal management DB 9002 (FIG.24B) using each of the terminal IDs read in step S107 as a search key,to read the terminal name, operating state information, and site name ofthe associated destination candidate (step S108).

Next, the data exchange unit 91 transmits destination list contentinformation to the display terminal 50A through the communicationnetwork 9 (step S109). The destination list content information includesthe terminal IDs of the destination candidates read in step S107 and theterminal names, operating state information, and site names of thedestination candidates read in step S108. Thereby, the data exchangeunit 51 of the display terminal 50A receives the destination listcontent information.

Next, the display control unit 54 of the display terminal 50A controlsthe display 511 to display the destination list screen 900, which isgenerated using the destination list frame data stored in the storageunit 5000 in step S105 and the destination list content informationreceived in step S109 (step S110). FIG. 27 is a diagram illustrating anexample of a destination list screen 900 displayed on the displayterminal, according to an embodiment. The destination list screen 900illustrated in FIG. 27, displays, for each of the destinationcandidates, an icon indicating the operating state of the destinationcandidate terminal (robot 10), the terminal ID of the destinationcandidate terminal, the destination name of the destination candidate,and the base name of the destination candidate terminal. Note that the“terminal name” received in step S109 is displayed as the “destinationname” on the destination list screen 900 illustrated in FIG. 27.

A description is now given of operation from a selection of adestination candidate at the display terminal 50 to a start oftransmission/reception of image data, with reference to FIG. 28. FIG. 28is a sequence diagram illustrating an example of operation from aselection of a destination candidate to a start of datatransmission/reception, performed by the remote control system 1 a,according to an embodiment.

First, the acceptance unit 52 of the display terminal 50A receivesselection of a destination candidate (in this example, the robot 10A-1)on the destination list screen 900 illustrated in FIG. 27 from the user(step S111). Then, the data exchange unit 51 transmits start requestinformation indicating a request for a start of transmission/receptionof image data, etc. to the communication management server 90 (stepS112). This start request information includes the terminal ID of thedisplay terminal 50A and the terminal ID of the destination candidateterminal. Thereby, the data exchange unit 91 of the communicationmanagement server 90 receives the start request information.

Next, the determination unit 93 of the communication management server90 determines whether the terminal ID of the display terminal 50Areceived in step S112 is stored in the session management table (seeFIG. 25B) (S113). In this example, the following describes a case inwhich the terminal ID of the destination candidate terminal (robot10A-1) is not stored.

When the terminal ID of the destination candidate terminal is notstored, the generation unit 94 generates a new session ID (step S114).Then, the storage/reading unit 99 adds, in the session management table(FIG. 25B), a record associating the session ID generated in step S114,and the terminal ID of the display terminal 50A and the terminal ID ofthe destination candidate terminal received in step S112 (step S115). Inthis example, as illustrated in FIG. 25B, a new record associating thesession ID “se3” with the terminal IDs “o01” and “rA01” are stored.

Next, the data exchange unit 91 transmits session start requestinformation requesting a start of a session to the display terminal 50A(step S116). This session start request information includes the sessionID generated in step S114. Thereby, the data exchange unit 51 of thedisplay terminal 50A receives the session start request information.

The storage/reading unit 99 of the communication management server 90searches the terminal management DB 9002 (see FIG. 24B) using theterminal ID of the destination candidate terminal (robot 10A-1) receivedin step S112 as a search key, to read the associated IP address (stepS117). Then, the data exchange unit 91 transmits session start requestinformation requesting a start of a session to the destination candidateterminal (robot 10A-1) indicated by the IP address read in step S117(step S118). This session start request information includes the sessionID generated in step S114. Thereby, the data exchange unit 31 of thedestination terminal (robot 10A-1) receives a session start instruction.

Thus, the source terminal (display terminal 50A) and the destinationterminal (robot 10A-1) each establish a communication session with thecommunication management server 90 (steps S119-1 and S119-2). In thefollowing description, it is assumed that the display terminal 50 usesthe communication session established with the communication managementserver 90 to perform streaming communication with the robot 10.

Operation Using Remote Communication:

Transmission of Screen Data and Display:

A description is now given of data transmitted from the robot 10 to thedisplay terminal 50 using the session established with the communicationmanagement server 90, and operation of controlling processing oroperation of the robot 10 by the display terminal 50. FIG. 29 is asequence diagram illustrating an example of operation of transmittingvarious data from the robot 10 to the display terminal 50 in the remotecontrol system 1 a, according to an embodiment.

First, the data exchange unit 31 of the robot 10 transmits, to thedisplay terminal 50, spherical image data acquired by the special imagecapturing device 21 and detailed image acquired by the generic imagecapturing device 24, using the communication session established withthe communication management server 90 (steps S11-1, S11-2). Thereby,the data exchange unit 51 of the display terminal 50 receives thespherical image data and the detailed image data transmitted from therobot 10.

The robot 10 starts image capturing by the special image capturingdevice 21 and the generic image capturing device 24, triggered by animage capturing instruction from the image capturing control unit 37 tothe special image capturing device 21 and the generic image capturingdevice 24. The captured image acquisition unit 38 of the robot 10acquires spherical image data, which is data of a spherical imageobtained by the special image capturing device 21, and detailed imagedata, which is data of a detailed image obtained by the generic imagecapturing device 24, respectively from the special image capturingdevice 21 and the generic image capturing device 24. The data exchangeunit 31 of the robot 10 transmits the spherical image data and thedetailed image data acquired by the captured image acquisition unit 38to the display terminal 50.

Next, the display screen generation unit 53 of the display terminal 50generates a display screen 600 a to be displayed on the display 511using the spherical image data and the detailed image data received bythe data exchange unit 51 (step S12). Then, the display control unit 54controls the display 511 to display the display screen 600 a generatedby the display screen generation unit 53 (step S13). Referring to FIG.30, the display screen 600 a displayed on the display 511 in step S13 isdescribed. The display screen 600 a illustrated in FIG. 30 includes adisplay area 630 and a display area 650. In the display area 630, thedetailed image represented by the detailed image data transmitted fromthe robot 10 is displayed. In the display area 650, the spherical imagerepresented by the spherical image data transmitted from the robot 10 isdisplayed. The display screen 600 a further includes a movementinstruction keypad 605, a zoom bar 611, and a speed bar 613. Themovement instruction keypad 605 is pressed to request movement of therobot 10 in the horizontal direction (forward, backward, right rotation,left rotation). The zoom bar 611 indicates a zoom state of the sphericalimage displayed in the display area 610. The speed bar 613 displays amovement speed indicating a state of the movement speed of the robot 10.

In the display area 650, a part (e.g., the viewable-area image Qillustrated in FIG. 17, FIG. 18A and FIG. 18B) of a spherical imagerepresented by the spherical image data transmitted from the robot 10 isdisplayed. The operator moves the mouse 500 to the display area 610 inwhich the spherical image is displayed and performs a predeterminedinput operation, to change the display direction of the spherical imageor the degree of zoom.

This enables the operator, who remotely operates the robot 10 using thedisplay terminal 50, to check a situation of the site where the robot 10is located while viewing the display screen 600 a on which the sphericalimage and the detailed image are displayed.

Since the operator of the robot 10 remotely operates the robot 10 whileviewing the spherical image and the detailed image displayed on thedisplay terminal 50, an image displayed on the display terminal 50 isrequired to be in real time. Therefore, the data exchange unit 31 of therobot 10 continuously transmits the spherical image data acquired by thespecial image capturing device 21 to the display terminal 50. Due tosuch characteristics, it is preferable that the spherical image dataobtained by the special image capturing device 21 is moving image data.In this case, the operator who operates the robot 10 using the displayterminal 50 operates the robot 10 remotely, while checking moving imagedata of the spherical image which is transmitted from the robot 10 andthereby streamed on the display terminal 50. This allows the operator tocheck a wide area around the robot 10 without changing the direction ofthe special image capturing device 21 or the robot 10.

Although the description given referring to FIG. 30 is of an example inwhich in which the movement of the robot 10 is remotely controlled byreceiving the operation input to the movement instruction keypad 605displayed on the display screen 600 a of FIG, the movement operation isnot limited to this. In another example, the movement operation of therobot 10 is performed by a keyboard, a dedicated controller such as agame pad having a joystick, or the like. In still another example, thedisplay terminal 50 displays a list of names of a destination to whichthe robot 10 is requested to be moved or a list of persons to meet onthe display screen 600 a, to allow the user to select a desired name orinput an address. In this case, the display terminal 50 transmits arequest command for moving the robot 10 to the input specific locationto the robot 10, and the robot 10 autonomously moves to the destinationspecified by the request command.

Referring again to FIG. 29, the robot 10 moves within the site based ona request command or the like described below transmitted from thedisplay terminal 50 (step S14). In this case, the movement control unit41 of the robot 10 controls the drive of the movement mechanism 17 basedon the request command or the like described below transmitted from thedisplay terminal 50.

Next, the state information generation unit 35 generates the stateinformation 170 indicating the state of the robot 10 based on the drivestate of the movement mechanism 17 acquired from the movement controlunit 41 (step S15). FIG. 31 is a diagram illustrating an example of thestate information 170 indicating the state of the robot 10, according toan embodiment. The state information 170 illustrated in FIG. 31 includesinformation corresponding to items of the drive direction (horizontalangle and vertical angle) and drive speed of the robot 10. The stateinformation 170 illustrated in FIG. 31 further includes informationcorresponding to items of the imaging direction (horizontal angle andvertical angle) of the generic image capturing device 24 and the zoomamount. The state information 170 indicates, a variable namecorresponding to each of the items and a numerical value correspondingto the variable name. For example, in the case of the state information170 illustrated in FIG. 31, the robot 10 is moving in the drive speed of“3.0 km/h” and the drive direction of the horizontal angle “30°” andvertical angle “45°”. Further, the imaging direction of the genericimage capturing device 24 is the horizontal angle “−22.2°” and verticalangle “10.00°”, and the zoom amount is “22.3%”. The items of the stateof the robot 10 included in the state information 170 is not limited tothe above described items. In another example, the items includeinformation such as a moving distance of the robot 10.

The data exchange unit 31 of the robot 10 transmits the stateinformation 170 generated by the state information generation unit 35 tothe display terminal 50 using the communication session established withthe communication management server 90 (step S16-1, step S16-2).Thereby, the data exchange unit 51 of the display terminal 50 receivesthe state information 170 transmitted from the robot 10.

Then, the storage/reading unit 59 of the display terminal 50 stores thestate information 170 received by the data exchange unit 51 in the stateinformation management DB 5001 (see FIG. 22A) stored in the storage unit5000 (step S17). Specifically, the storage/reading unit 59 stores eachof the numerical values included in the received state information 170in a field associated with the corresponding variable name of the stateinformation management table, thereby updating the value of each itemincluded in the state information management table.

Next, the display screen generation unit 53 generates a display screen600 b to be displayed on the display 511 using the state information 170received in step S16-2 to (step S18). The display screen 600 b generatedby the display screen generation unit 53 is screen data in which thecharacteristics of the received state information 170 is includedtogether with the spherical image data and the detailed image datatransmitted from the robot 10. Then, the display control unit 54controls the display 511 to display the display screen 600 b generatedby the display screen generation unit 53 (step S19).

A description is now given of the display screen 600 b displayed in stepS19, with reference to FIG. 32. FIG. 32 is an illustration of an exampleof a display screen displayed on the display terminal 50 when the robot10 is moving forward in the remote control system 1 a, according to anembodiment. The display screen 600 b illustrated in FIG. 32 includesindications of the movement state of the robot 10 in addition to thedisplay screen 600 a. The display screen 600 b includes an object 651 ahaving a cylindrical shape, a drive direction icon 651 b, and an imagingposition icon 653 in the display area 650 in which the spherical imageis displayed. The object 651 a indicates the position of the robot 10.The drive direction icon 651 b has a shape of arrow indicating adirection in which the robot 10 is moving. The imaging position icon 653indicates an imaging position of the generic image capturing device 24.The object 651 a is an example of an object image, and the drivedirection icon 651 b is an example of a drive direction image. Further,the imaging position icon 653 is an example of an imaging positionimage.

Further, on the display screen 600 b, the colors of the “forward” and“left rotation” keys of the movement instruction keypad 605 are changed,thereby indicating that the robot 10 is moving forward to the left. Thedisplay screen generation unit 53 updates the indication of the movementinstruction keypad 605 based on the numerical values corresponding tothe variable names “DRIVE_H_ANGLE” and “DRIVE_V_ANGLE” included in thestate information 170. In another example, the display screen generationunit 53 updates the indication of the movement instruction keypad 605 inresponse to receiving an operation input on the movement instructionkeypad 605. Further, on the display screen 600 b, the drive speed of therobot 10 is indicated by the ratio of the black portion of the speed bar613. The display screen generation unit 53 updates the display of thespeed bar 607 based on the numerical value corresponding to the variablename “DRIVE_SPEED” included in the state information 170.

A description is now given of a spherical image displayed on the displayterminal 50, with reference to FIG. 33. FIG. 33 is a diagram fordescribing an example of a spherical image displayed on the displayterminal 50, according to an embodiment. As illustrated in FIG. 16A andFIG. 16B to FIG. 19, the spherical image CE is projected on the innerwall of the three-dimensional virtual space (the solid sphere CS) havinga spherical shape, and the virtual camera IC is arranged at the centerposition of the three-dimensional virtual space. An image viewed fromthe virtual camera IC is an image displayed on the display 511 of thedisplay terminal 50.

As illustrated in FIG. 33, in the present embodiment, the virtual cameraIC arranged inside the solid sphere CS is arranged as being shiftedupward and rearward from the center position of the sphere CS. Further,the object 651 a indicating the robot 10 is arranged at the centerposition of the three-dimensional virtual space (solid sphere CS) havinga spherical shape. The display screen generation unit 53 superimposes animage indicating the presence of the robot 10, i.e., the object 651 a,on the spherical image CE viewed from the virtual camera IC, i.e. theviewable-area image Q, to generate the display screen 600 b.

Thus, the display terminal 50 arranges the virtual camera IC at aposition apart from the center position of the three-dimensional virtualspace (solid sphere CS) having a spherical shape, and sets the positionwhere the virtual camera IC is arranged as the origin of the field ofview, thereby bringing the existence of the robot 10 (the object 651 a)within the field of view of the virtual camera IC. This enables the userof the display terminal 50 to check both the existence of the robot 10(the object 651 a) and the spherical image CE (the viewable-area imageQ) at the same time from a so-called pseudo third-person view (TPV).

Further, when changing the display position (the viewable area T) of thespherical image CE, the display terminal 50 changes the position of thevirtual camera IC, which is the origin of the field of view. In thiscase, when changing the position of the virtual camera IC, the displayterminal 50 moves the position of the virtual camera IC so that theexistence of the robot (the object 651 a) is always within the view ofthe virtual camera IC. For example, when changing the display position(the viewable area T) of the spherical image CE in the verticaldirection (upper side of the solid sphere CS illustrated in FIG. 33),the display terminal 50 not only changes the direction of the virtualcamera IC (changes the field of view upward), but also moves theposition of the virtual camera IC in the direction opposite with respectto the field of view of the virtual camera IC, i.e., moves the positionof the virtual camera IC to the lower side of the sheet on which FIG. 33is drawn. Further, for example, when changing the display position (theviewable area T) of the spherical image CE from left to right (rightside of the solid sphere CS illustrated in FIG. 33), the displayterminal 50 not only changes the direction of the virtual camera IC(changes the field of view to the right), but also moves the position ofthe virtual camera IC in the direction opposite with respect to thefield of view of the virtual camera IC, i.e., moves the position of thevirtual camera IC to the left side of the sheet on which FIG. 33 isdrawn. As a result, the virtual camera IC brings the presence of therobot (the object 651 a) into view even when the display position (theviewable area T) of the spherical image is changed.

Thus, by checking the spherical image CE displayed on the display 511,the user of the display terminal 50 smoothly recognizes the situation ofthe site as well as the relative position of the robot 10 by the object651 a. Accordingly, the operability of the robot 10 is improved.

Further, as illustrated in FIG. 33, the drive direction icon 651 bindicating the drive direction of the robot 10 is arranged inside thesolid sphere CS. The display screen generation unit 53 generates thedisplay screen 600 b in which the drive direction icon 651 b issuperimposed on the spherical image CE (the viewable-area image Q),based on the numerical values corresponding to the variable names“DRIVE_H_ANGLE” and “DRIVE_V_ANGLE” included in the state information170. Thereby, even when the user of the display terminal 50 changes thedisplay position (the viewable area T) of the spherical image CE by, forexample, an operation of changing the display direction of the sphericalimage or enlarging or reducing the spherical image, the drive directionicon 651 b is displayed on the display screen 600 b. This enables theuser to recognize the state (direction or drive direction) of the robot10, thereby improving the operability of the robot 10.

Further, as illustrated in FIG. 33, the imaging position icon 653indicating the imaging position of the generic image capturing device 24is arranged inside the solid sphere CS. The display screen generationunit 53 generates the display screen 600 b in which the imaging positionicon 653 is superimposed on the spherical image CE (the viewable-areaimage Q), based on the numerical values corresponding to the variablenames “CAMERA_H_ANGLE” and “CAMERA_V_ANGLE” included in the stateinformation 170. This enables the user of the display terminal 50 torecognize which position in the spherical image CE (the viewable-areaimage Q) is imaged to obtain the detailed image displayed on the displayscreen 600 b.

Thus, the display terminal 50 generates the pseudo TPV by using thespherical image CE, and displays not only the situation of the siteindicated by the spherical image CE, but also the position and movementstate of the robot 10 and the imaging position by the generic imagecapturing device 24, each being superimposed on the spherical image,thereby improving operability while allowing the user to recognize thesituation of the site.

As described heretofore, the remote control system 1 a controls thedisplay terminal 50 to display the spherical image and the detailedimage based on the spherical image data and detailed image datatransmitted from the robot 10 (control device 30), thereby enabling theoperator who remotely operates the robot 10 using the display terminal50 to more accurately recognize information around the robot 10.Further, the display terminal 50 displays information indicating themovement state of the robot 10 together with the spherical imagerepresented by the spherical image data, thereby notifying the operatorwho remotely operates the robot 10 using the display terminal 50 of themovement state of the robot 10 in addition to information around therobot 10.

In one example, the state information generation unit 35 of the robot 10is configured to generate the state information 170 when the drive stateof the movement mechanism 17 acquired from the movement control unit 41changes. In another example, the state information generation unit 35 isconfigured to generate the state information 170 at intervals determinedin advance. In still another example, the data exchange unit 31 of therobot 10 is configured to hold the state information 170 generated bythe state information generation unit 35 and transmit the stateinformation 170 at intervals determined in advance. In this case, theremote control system 1 a can reduce uncomfortable feelings of the userviewing the display screen caused by the frequent change of the state ofthe robot 10 displayed on the display terminal 50.

Remote Control of Robot:

A description is now given of remote control of the robot 10 performedusing the display screen displayed on the display terminal 50. First,referring to FIG. 34, an example of remote control of the robot 10 basedon the state information 170 transmitted from the robot 10 is described.FIG. 34 is a flowchart illustrating an example of operation ofcontrolling the robot 10 based on the movement state of the robot 10performed by the display terminal 50, according to an embodiment.

When the data exchange unit 51 receives the state information 170transmitted from the robot 10 (YES in step S31 a), the operationproceeds to step S32 a. By contrast, when the data exchange unit 51receive no state information 170 (NO in step S31 a), the process of stepS31 a is repeated.

The storage/reading unit 59 reads the condition information stored inthe condition information management DB 5002 (see FIG. 22B) (step S32a). Next, the determination unit 56 searches the condition informationread by the storage/reading unit 59, to determine whether there is anyitem satisfying the conditions indicated in the condition informationamong the variables indicated in the state information 170 received instep S31 a (step S33 a). Then, when the determination unit 56 determinesthat there is any item satisfying the condition indicated in thecondition information among the variables indicated in the stateinformation 170 (YES in step S33 a), the operation proceeds to step S34a. For example, when the value associated with the variable name“DRIVE_SPEED” included in the state information 170 is “3.0 km/h”, thedetermination unit 56 determines that among the items indicated in thecondition information of FIG. 22B, “SPEED≤5.0 km/h”, “SPEED>1.0 km/h”and “SPEED>2.0 km/h” satisfy the conditions. By contrast, when thedetermination unit 56 determines that there is no item satisfying thecondition indicated in the condition information among the variablesindicated in the state information 170 (NO in step S33 a), the operationends.

Next, the request command generation unit 57 identifies the processingassociated with the item identified as satisfying the condition in stepS33 a, to generate a request command (step S34 a). For example, therequest command generation unit 57 identifies the processingcorresponding to each of “SPEED≤5.0 km/h”, “SPEED>1.0 km/h”, and“SPEED>2.0 km/h” satisfying the conditions indicated in the conditioninformation. Then, the request command generation unit 57 generates arequest command using the commands each corresponding the identifiedprocessing indicated in the condition information. In this case, thecommands included in the request command are “FPS (30)”, “RESOLUTION(50%)”, and “ZOOM=10”. In one example, the request command includes allthe commands corresponding to the items satisfying the conditions. Inanother example, the request command includes only the commandcorresponding to an item selected (extracted) from the items satisfyingthe condition.

Then, the data exchange unit 51 transmits the request command generatedby the request command generation unit 57 to the robot 10 using thecommunication session established with the communication managementserver 90 (step S35 a). Thus, the data exchange unit 31 of the robot 10receives the request command corresponding to the current movement stateof the robot 10 from the display terminal 50. In response to receivingthe request command, the robot 10 executes command processing describedbelow with reference to FIG. 36.

As described heretofore, the display terminal 50 requests a change inthe image quality of the spherical image acquired by the special imagecapturing device 21 based on the movement state of the robot 10, tochange the image quality of the spherical image to be displayed on thedisplay terminal 50. For example, when the robot 10 is moving, thedisplay terminal 50 may send a request to the robot 10 to reduce theimage quality of the spherical image. Specifically, the display terminal50 transmits a request to the robot 10 to reduce the resolution of thespherical image or reduce the frame rate. Thus, the remote controlsystem 1 a reduces the resolution or the update frequency (frame rate)of the spherical image displayed on the display terminal 50. Thisreduces uncomfortable feeling such as motion sickness that the operatorwho operates the robot 10 using the display terminal 50 feels whenoperating the robot 10 while viewing the spherical image.

Further, when performing the remote control of the robot 10 using thedisplay terminal 50, the operator performs an operation input whileviewing the spherical image displayed on the display terminal 50, andthus the spherical image transmitted from the robot 10 is required to bein real time. When a communication delay occurs due to congestion of thecommunication network 9, etc., a time lag occurs between the sphericalimage displayed on the display terminal 50 and an actual situationaround the robot 10, thereby reducing operability for the operator whoremotely operates the robot 10. To address such issue, the remotecontrol system 1 a changes the image quality of the spherical imagetransmitted from the robot 10 according to the movement state of therobot 10, thereby preventing deterioration in operability caused by thetime lag on the display screen due to communication delay or the like.

A description is now given of an example of remote control of the robot10 based on an input command input by an operator who operates the robot10 using the display terminal 50, with reference to FIG. 35. FIG. 35 isa flowchart illustrating an example of operation of controlling therobot 10 based on an input command, performed by the display terminal50, according to an embodiment. In the following, an example case isdescribed in which the display screen 600 a (see FIG. 30) is displayedon the display 511 of the display terminal 50, with reference to FIG.35. Note that the display screen 600 b (see FIG. 32) may be displayed onthe display 511 of the display terminal 50.

When the acceptance unit 52 receives a particular operation input on thedisplay screen 600 a displayed on the display 511 (YES in step S31 b),the operation proceeds to step S32 b. In the embodiment, the particularoperation input is an operation input to any one of the movementinstruction keypad 605 displayed on the display screen 600 a. Bycontrast, when the acceptance unit 52 receives no particular operationinput to the display screen 600 a (NO in step S31 b), the process ofstep S31 b is repeated.

The storage/reading unit 59 searches the user command management DB 5003(see FIG. 23A) using a content of the particular operation inputreceived in step S31 b as a search key, to extract the associatedprocessing (step S32 b). Specifically, the storage/reading unit 59searches the user command management DB 5003 to identify an inputcommand corresponding to the operation input received by the acceptanceunit 52. Then, the storage/reading unit 59 extracts processingassociated with the identified input command. For example, when theacceptance unit 52 receives an input to the “↑ (forward)” key of themovement instruction keypad 605, the storage/reading unit 59 identifies“PRESS “↑ (FORWARD)” KEY” as the input command. Then, thestorage/reading unit 59 extracts “MOVE (10.0, 10.0)” as the processingassociated with the identified input command.

The determination unit 56 further determines whether the type ofprocessing associated with the processing extracted in step S32 b is“TRANSMIT REQUEST COMMAND” (step S33 b). When the determination unit 56determines that the type of processing associated with the extractedprocessing is “TRANSMIT REQUEST COMMAND” (YES in step S33 b), theoperation proceeds to step S34 b. Then, the request command generationunit 57 generates the request command corresponding to the processingextracted by the determination unit 56 (step S34 b). For example, whenthe extracted processing is “MOVE (10.0, 10.0)”, the request commandgeneration unit 57 generates a request command including “MOVE (10.0,10.0)”. Then, the data exchange unit 51 transmits the request commandgenerated by the request command generation unit 57 to the robot 10using the communication session established with the communicationmanagement server 90 (step S35 b). Thus, the data exchange unit 31 ofthe robot 10 receives the request command corresponding to the inputoperation by the operator from the display terminal 50. In response toreceiving the request command, the robot 10 executes command processingdescribed below with reference to FIG. 36.

By contrast, when the determination unit 56 determines that the type ofprocessing associated with the extracted processing is not “TRANSMITREQUEST COMMAND” in step S33 b, the operation proceeds to step S36 b.The display terminal 50 performs the processing extracted in step S32 b(step S36 b). For example, when the extracted processing is “LOOK (H, V,Z)” based on “MOUSE MOVEMENT”, the display terminal 50 changes thedisplay position of the spherical image displayed in the display area650 by the display control unit 54.

The description given above is of an example in which the processing tobe executed is extracted by using the user command stored in the usercommand management DB 5003 in step S32 a. In another example, thedisplay terminal 50 is configured to extract or execute predeterminedprocessing based on, for example, an event given to the movementinstruction keypad 605.

As described heretofore, the display terminal 50 requests a change inthe image quality of the spherical image acquired by the special imagecapturing device 21 based on the movement state of the robot 10, tochange the image quality of the spherical image to be displayed on thedisplay terminal 50. For example, when the robot 10 is moving, thedisplay terminal 50 may send a request to the robot 10 to reduce theimage quality of the spherical image. Specifically, the display terminal50 transmits a request to the robot 10 to reduce the resolution of thespherical image or reduce the frame rate. Thus, the remote controlsystem 1 a reduces the resolution or the update frequency (frame rate)of the spherical image displayed on the display terminal 50. Thisreduces uncomfortable feeling such as motion sickness that the operatorwho operates the robot 10 using the display terminal 50 feels whenoperating the robot 10 while viewing the spherical image.

Referring to FIG. 36, a description is now given of operation performedby the robot 10 using the request command transmitted from the displayterminal 50 by the operation illustrated in FIG. 34 or FIG. 35. FIG. 36is a flowchart illustrating an example of operation of controlling therobot 10 based on the request command from the display terminal 50,performed by the robot 10, according to an embodiment.

When the data exchange unit 31 receives the request command transmittedfrom the display terminal 50 (YES in step S51), the operation proceedsto step S52. By contrast, when the data exchange unit 31 receives norequest command (NO in step S51), the process of step S51 is repeated.

The storage/reading unit 39 searches the command management DB 3001 (seeFIG. 21A) using the command indicated in the request command received bythe data exchange unit 31 as a search key, to extract correspondingprocessing (step S52). Then, the robot 10 performs the processingextracted in step S52 (step S53).

For example, when the request command received by data exchange unit 31is “MOVE (10.0, 10.0)”, the determination unit 34 searches for the readcommand to extract processing corresponding to the command name “MOVE”.In this case, the extracted processing is “Rotate left wheel by 10.0°,Rotate right wheel by 10.0°”. The determination unit 34 send anotification indicating an execution request of the extracted processingto the movement control unit 41. Then, the movement control unit 41rotates the left wheel of the movement mechanism 17 of the robot 10 by10.0° and rotates the right wheel by 10.0° based on the processingextracted by the determination unit 34. Note that a destination to whichthe notification indicating the execution request is sent variesdepending on the processing extracted by the determination unit 34. Forexample, when the request command received by the data exchange unit 31is “AIM”, “FPS”, or “RESOLUTION”, the notification destination of theexecution request is the image capturing control unit 37.

As described heretofore, the robot 10 performs processing or anoperation based on the request command transmitted from the displayterminal 50 by the operation illustrated in FIG. 34 or FIG. 35, and alsotransmits an image (the spherical image data or the detailed image data)captured and acquired based on the request command to the displayterminal 50. Thereby, the remote control system 1 a implements remoteoperation the robot 10 using the display terminal 50, and also controlsthe display terminal 50 used by the operator of the robot 10 to displayinformation (e.g., the spherical image data of the detailed image data)that the robot 10 acquires as a result of the remote operation.

Display Control at Display Terminal:

Change of Display Position of Spherical Image:

A description is now given of display of the spherical image and thedetailed image to the display terminal 50, with reference to FIG. 37 toFIG. 41. First, an operation of changing a display position of thespherical image displayed on the display terminal 50 is described withreference to FIG. 37. FIG. 37 is a flowchart illustrating an example ofoperation of changing the display position of the spherical image on thedisplay terminal 50, according to an embodiment. With reference to FIG.37, an example case is described in which a display screen 600 cillustrated in FIG. 39A is displayed on the display 511 of the displayterminal 50. Unlike the display screen 600 b illustrated in FIG. 32, thedisplay screen 600 c illustrated in FIG. 39A includes only the displayarea 650 displaying the spherical image. The display terminal 50 canswitch the display of the spherical image and the detailed image by adisplay screen switching operation (see FIG. 38 and FIG. 39B) describedbelow.

A user who uses the display terminal 50 moves the mouse 500 connected tothe display terminal 50 via the external device connection I/F 513 tochange the display position of the spherical image displayed on thedisplay 511. First, when the acceptance unit 52 of the display terminal50 detects a movement of the mouse 500 (YES in step S61), the operationproceeds to step S62. By contrast, when the acceptance unit 52 detectsno movement of the mouse 500 (NO in step S61), the process of step S61is repeated.

Next, the display position identification unit 55 calculates an amountof the movement of the mouse 500 (step S62). Then, the display controlunit 54 changes the display position of the spherical image displayed inthe display area 650 based on the movement amount calculated by thedisplay position identification unit 55 (step S63). Specifically, thedisplay control unit 54 displays, in the display area 650, a position(area) that is distant from the center of the display position (e.g.,the viewable area T (e.g., a first area) illustrated in FIG. 33) of thespherical image by a distance corresponding to the movement amountcalculated by the display position identification unit 55, as a viewablearea T (e.g., a second area).

Conventionally, when changing the display position of the display screenwith the mouse, the position of the mouse cursor displayed on thedisplay is detected and the display position is changed according to themovement amount of the mouse cursor. However, with this method, when themouse cursor reaches the edge of the display screen, the cursor cannotbe moved any further. Therefore, a user is required to release themovement once, return the moue cursor to the center of the displayscreen, and move the mouse cursor again. For the user, the operabilitywas poor, and the operation was time consuming. To address such issue,the display terminal 50 changes the display position of the sphericalimage according to the movement amount of the mouse 500 instead of themovement amount of the mouse cursor, thereby smoothly displaying animage of the position that the user wants to view independent of thescreen size or the like of the display 511. Thus, the display terminal50 improves the user's operability for the displayed spherical image.

The determination unit 56 determines whether the movement of the mouse500 detected in step S61 has stopped (step S64). For example, when atime period from when the movement of the mouse 500 is detected by theacceptance unit 52 to when the movement of the mouse 500 is no longerdetected exceeds a predetermined time period, the determination unit 56determines that the movement of the mouse 500 has stopped. When thedetermination unit 56 determines that the acceptance unit 52 detects thestop of the mouse 500 (YES in step S64), the operation proceeds to stepS65. Then, when the stop of the mouse is detected, the request commandgeneration unit 57 generates a request command including “AIM (H, V, Z)”(step S65). Then, the data exchange unit 51 transmits the requestcommand generated in step S65 to the robot 10 using the communicationsession established with the communication management server 90 (stepS66). Although the request command may be sent to the robot 10 everytime the mouse 500 moves, in such case, the request command is sent toofrequently. Therefore, in the embodiment, the request command istransmitted when the mouse 500 does not move for the predetermined timeafter the detection of the movement.

By contrast, in step S64, when the determination unit 56 determines thatthe acceptance unit 52 does not detect the stop of the mouse 500 (NO instep S64), the operation proceeds to step S67. When the stop of themouse is not detected, the request command generation unit 57 generatesa request command including “LOOK (H, V, Z)” (step S67). Then, the dataexchange unit 51 transmits the request command generated in step S67 tothe robot 10 using the communication session established with thecommunication management server 90 (step S68). Further, the displayterminal 50 repeats the processes from step S62 until the movement ofthe mouse 500 is stopped (YES in step S64).

Thus, the display terminal 50 changes an area (the viewable area T),which is a part of the spherical image, to be displayed on the display511 according to the movement of the mouse 500, thereby displaying animage indicating a different area on the display 511. This improves theoperability when the user checks the situation of the remote site whileviewing the spherical image.

Switching of Display Image:

Next, with reference to FIG. 38, a description is given of an operationperformed when the request command generated by the operationillustrated in FIG. 37 is transmitted to the robot 10. FIG. 38 is asequence diagram illustrating an example of an operation of changing animaging position, performed by the generic image capturing device 24,according to an embodiment. FIG. 38 illustrates an operation oftransmitting the “AIM” command illustrated in step S66 of FIG. 37 fromthe display terminal 50 to the robot 10.

First, the data exchange unit 51 of the display terminal 50 transmits arequest command including “AIM” generated by the request commandgeneration unit 57 to the robot 10 (steps S81-1, S81-2). Thereby, thedata exchange unit 31 of the robot 10 receives the request commandtransmitted from the display terminal 50.

Next, the storage/reading unit 39 of the robot 10 searches the commandmanagement DB 3001 (see FIG. 21A) using the command indicated in therequest command received by the data exchange unit 31 as a search key,to read corresponding processing (step S82). In this case, thestorage/reading unit 39 reads the processing associated with the command“AIM”. Then, the image capturing control unit 37 changes the imagingposition of the generic image capturing device 24 based on theprocessing read in step S82 (step S83). Specifically, the imagecapturing control unit 37 transmits, to the generic image capturingdevice 24, instruction information indicating that the imaging directionis to be moved so that the horizontal angle is H degrees and thevertical angle is V degrees with respect to the front of the robot 10and the zoom amount is changed to Z %. The generic image capturingdevice 24 performs photographing with the parameters (H, V, Z) indicatedin the instruction information transmitted from the image capturingcontrol unit 37, to acquire a detailed image. When the generic imagecapturing device 24 is the movable camera 23 (see FIG. 3 and FIG. 4A toFIG. 4C), the generic image capturing device 24 rotates by using thebuilt-in movement mechanism to change the imaging position toward theparameters (H, V, Z) indicated in the instruction information. When thegeneric image capturing device 24 is the normal camera 25 (see FIG. 5Aand FIG. 5B to FIG. 9A and FIG. 9B), the imaging position by the genericimage capturing device 24 is changed toward the parameters (H, V, Z)indicated by the instruction information by rotation and/or deformationof the movable arm 11.

Next, the state information generation unit 35 generates the stateinformation 170 indicating the state of the robot 10 based on theinstruction information transmitted by the image capturing control unit37 (step S84). In this case, the state information generation unit 35generates the state information 170 indicating the parameters regardingthe imaging position of the detailed image acquired by the generic imagecapturing device 24.

Then, the data exchange unit 31 transmits the state information 170generated in step S84 and the detailed image data captured by thegeneric image capturing device 24 to the display terminal 50 using thecommunication session established with the communication managementserver 90 (step S85-1, step S85-2). Thereby, the data exchange unit 51of the display terminal 50 receives the state information 170 and thedetailed image data transmitted from the robot 10.

Then, the storage/reading unit 59 of the display terminal 50 stores thestate information 170 received by the data exchange unit 51 in the stateinformation management DB 5001 (see FIG. 22A) stored in the storage unit5000 (step S86). Specifically, the storage/reading unit 59 stores eachof the numerical values included in the received state information 170in a field associated with the corresponding variable name of the stateinformation management table, thereby updating the value of each itemincluded in the state information management table.

Next, the display screen generation unit 53 generates a display screen600 d to be displayed on the display 511 using the state information 170received in step S16-2 to (step S87). Specifically, first, thestorage/reading unit 59 reads out the condition information stored inthe condition information management DB 5002 (see FIG. 22B). Next, thedetermination unit 56 searches the condition information read by thestorage/reading unit 59. When the imaging position of the generic imagecapturing device 24 indicated by the numerical values (parameters) ofthe variable names “CAMERA_H_ANGLE”, “CAMERA_V_ANGLE” and “CAMERA_ZOOM”included in the state information 170 received in step S85-2 is close tothe display position of the spherical image, the determination unit 56determines that the corresponding condition is satisfied. Then, thedisplay screen generation unit 53 generates the display screen 600 ddisplaying the detailed image represented by the detailed image datareceived in step S85-2.

Although in the following, a description is given of the display screen600 d, when the determination unit 56 determines that there is no itemcorresponding to the searched condition information, that is, when thedetermination unit 56 determines that the imaging position of thegeneric image capturing device 24 is not close to the display positionof the spherical image, the display screen generation unit 53 generatesthe display screen 600 c displaying the spherical image as illustratedin FIG. 39A. In the embodiment, “the imaging position of the genericimage capturing device 24 is close to the display position of thespherical image” is not limited to that the imaging position of thegeneric image capturing device 24 and the display position of thespherical image are the same. It suffices that the positions are withina predetermined range. This predetermined range is set by a designer ora user, for example. In another example, the predetermined range isdetermined according to a parameter such as the degree of matching orthe degree of similarity between the displayed spherical image and thedetailed image photographed according to the imaging position of thegeneric image capturing device 24. In this case, for example, when theparameter such as the degree of matching or the degree of similaritybetween the spherical image and the detailed image is equal to or morethan a set value, the display terminal 50 determines that the imagingposition of the generic image capturing device 24 is close to thedisplay position of the spherical image. In addition, instead of thevariables indicated in the state information 170, the variablesindicated in the request command (“AIM”) transmitted in step S81 may beused as a prediction value of the imaging position of the generic imagecapturing device 24.

Then, the display control unit 54 controls the display 511 to displaythe display screen 600 d generated by the display screen generation unit53 (step S88). The display screen 600 d illustrated in FIG. 39B includesthe display area 630 displaying the detailed image, instead of thespherical image displayed on the display screen 600 c illustrated inFIG. 39A. As described heretofore, the display terminal 50 automaticallyswitches the display from the spherical image to the detailed image whenthe imaging position of the generic image capturing device 24 catches upto the display position of the spherical image.

Thus, the remote control system 1 a changes the imaging position of thegeneric image capturing device 24 so that the imaging position followsthe display position of the spherical image, based on the requestcommand that is transmitted from the display terminal 50 with the changeof the display position of the spherical image as a trigger. Further,the display terminal 50 displays, in place of the spherical image, thedetailed image acquired by the generic image capturing device 24 whoseimaging position is changed so that the imaging position follows thedisplay position of the spherical image. This enables the user to viewthe spherical image to recognize the situation of the remote site untilthe imaging position of the detailed image catches up to the displayposition of the spherical image, and view the detailed image torecognize more detailed information of the remote site when the imagingposition of the detailed image catches up to the display position of thespherical image. Further, the user of the display terminal 50 recognizesthe imaging position of the generic image capturing device 24 by viewingthe imaging position icon 653 displayed on the display screen 600 cuntil the imaging position of the generic image capturing device 24catches up to the display position of the spherical image. This enablesthe user to predict the timing when the spherical image is switched tothe detailed image. Thus, the display terminal 50 switches the displaybetween the spherical image and the detailed image according to thecondition, thereby allowing the user to check the state of the remotesite efficiently and with high accuracy. This improves the operabilityfor the user.

Further, when the robot 10 includes plural generic image capturingdevices 24 (movable cameras 23) as illustrated in FIG. 4B, the imagecapturing control unit 37 of the robot 10 can select the generic imagecapturing device 24 to which an image capturing instruction is to givenin accordance with the variables included in the request command. Forexample, when the display position of the spherical image is in theupward direction of the solid sphere CS, the image capturing controlunit 37 instructs the movable camera 23 a to perform image capturing. Onthe other hand, when the display position of the spherical image is inthe downward direction of the solid sphere CS, the movable camera 23 bis instructed to perform image capturing. The display terminal 50displays the detailed image acquired by either the movable camera 23 aor the movable camera 23 b. This enables the remote control system 1 ato reduce blind spots such as the ceiling or the floor in the imagecapturing by the generic image capturing device 24. In another example,the image capturing control unit 37 instructs both the movable camera 23a and the movable camera 23 b to perform image capturing, and thedisplay terminal 50 displays plural detailed images.

Conventionally, the special image capturing device 21 and the genericimage capturing device 24 are controlled independently. This requires auser to first view a desired direction of the spherical image acquiredby the special image capturing device 21 and thereafter to give an imagecapturing instruction to the generic image capturing device 24. Toaddress such issue, the remote control system 1 a controls the displayposition of the spherical image and the imaging position of the genericimage capturing device 24 to link with each other, thereby enabling theuser to seamlessly recognize the entire remote site and an area wherethe user has interest to details. This improves the operability whilealso improving the resolution of the display image.

The display screen 600 d may display the object 651 a indicating theposition of the robot 10 and/or the drive direction icon 651 bindicating the drive direction of the robot 10, in substantially thesame manner as the display screen 600 c, etc. In addition, although thedescription given above with reference to FIG. 39A and FIG. 39B is of anexample case in which the display on a single screen is switched betweenthe spherical image and the detailed image, no limitation is intendedthereby. In another example, the display terminal 50 displays both thespherical image and the detailed image like the display screen 600 aillustrated in FIG. 30 and the display screen 600 b illustrated in FIG.32. In this case, although the display terminal 50 does not switch thedisplay between the spherical image and the detailed image, the displayterminal 50 displays the detailed image acquired by the generic imagecapturing device 24 that changes the imaging position so that theimaging position follows the display position of the spherical image.Further, the description given above with reference to FIG. 38 is of anexample case in which the display is automatically switched between thespherical image and the detailed image, no limitation is intendedthereby. In another example, the display is switched between thespherical image and the detailed image in response to selection ofeither one of tabs (“SPHERICAL” or “DETAILED”) in the upper right of thedisplay area 630 or the display area 650.

A description is now given of another example of a display screendisplayed on the display terminal 50, with reference to FIG. 40. Adisplay screen 600 e illustrated in FIG. 40 is an example of a screendisplayed when the display position of the spherical image is a positionthat cannot be imaged by the generic image capturing device 24. Thegeneric image capturing device 24, either the movable camera 23 or thenormal camera 25, has a direction or range in which the generic imagecapturing device cannot move due to its structure. For example, thegeneric image capturing device 24 cannot photograph the direction ofdirectly behind, directly below such as a floor, or directly above suchas a ceiling. Therefore, the display screen 600 e includes an image 655indicating an area (immovable area) that cannot be photographed by thegeneric image capturing device 24 in the display area 650 displaying thespherical image. The display screen 600 e is an example in which thedisplay position of the spherical image is in the downward direction ofthe robot 10. The image 655 is an example of an unacquirable image,which indicates that the detailed image cannot be acquired.

Thus, when the display position of the spherical image is changed to aposition that cannot be imaged by the generic image capturing device 24,the display terminal 50 allows the user to intuitively recognize asituation where the spherical image cannot be switched to the detailedimage. Further, when the user of the display terminal 50 notices theimage 655 displayed on the display terminal 50, which indicates an area(immovable area) that cannot be captured by the generic image capturingdevice 24, the user smoothly consider an alternative operation to betaken, such as operation of moving the robot 10.

Another Example of Screen of Display Screen:

A description is now given of another example of the display screendisplayed on the display terminal 50. The display terminal 50 isconfigured to display different display screens according to the type ofthe display 511 that the display terminal 50 includes. FIG. 41 is adiagram illustrating an example of a display screen displayed on ahead-mounted display as an example of the display terminal 50, accordingto an embodiment. A display screen 600 f illustrated in FIG. 41 is anexample of a screen in which the spherical image data and the detailedimage data, each transmitted from the robot 10, are displayed on ahead-mounted display (HMD) as an example of a virtual reality (VR)terminal.

On the display screen 600 f illustrated in FIG. 41, the entire screen isdisplayed as the display area 650 for the spherical image, and thedisplay area 630 for the detailed image is superimposed thereon. Anoperator wearing a head-mounted display displaying the display screen600 f is able to always recognize the entire remote site by theperipheral vision, and to view a desired direction of the sphericalimage by moving his or her head. Further, since various buttons or iconsdisplayed on the display screen 600 f move following the orientation ofthe operator's head, the various buttons or icons are displayed atpredetermined positions even when the display position of the sphericalimage changes. Furthermore, when the head-mounted display is used, thedirection in which the operator is looking is acquired with a gyroscopicsensor, etc. This enables the display terminal 50 to feed back to therobot 10 the direction viewed by the operator as the display position ofthe spherical image, in substantially the same manner as describedabove. The display terminal 50 may display the display screen 600 f asillustrated in FIG. 41 using a wide-angle screen or an omnidirectionaldisplay, in addition the head-mounted display.

Display Screen Displayed at Robot:

A description is now given of a display screen displayed at the robot10, with reference to FIG. 42, FIG. 43A, and FIG. 43B. In the followingdescription, it is assumed that the robot 10 includes the display 150like the robot 10 c (10 c 1 to 10 c 3) illustrated in FIG. 10A to FIG.10C. FIG. 42 is a sequence diagram illustrating an example of operationof causing the robot 10 to display a captured image acquired by thedisplay terminal 50, performed by the remote control system 1 a,according to an embodiment.

First, the data exchange unit 51 of the display terminal 50 transmitscaptured-image data acquired by the CMOS sensor 505 a to the robot 10using a communication session established with the communicationmanagement server 90 (step S151-1, step S151-2). In the embodiment, theCMOS sensor 505 a of the display terminal 50 captures an image of theoperator who is remotely operating the robot 10 using the displayterminal 50. Thereby, the data exchange unit 31 of the robot 10 receivesthe captured-image data transmitted from the display terminal 50.

Next, the storage/reading unit 39 of the robot 10 reads out the imagecapturing parameters stored in the image capturing parameter managementDB 3002 (step S152). The display control unit 54 generates the displayscreen 800 to be displayed on the display 150 by using thecaptured-image data received in step S151-2 and the image capturingparameter read in step S152 (step S153). Then, the display control unit54 controls the display 150 to display the display screen 800 generatedin step S153 (step S154).

A description is now given of the display screen 800 displayed on thedisplay 150 of the robot 10, with reference to FIG. 43A and FIG. 43B.The display screen 800 illustrated in FIG. 43A includes an image of theface of the operator who is remotely operating the robot 10 using thedisplay terminal 50. The user at the site where the robot 10 is locatedrecognizes who is operating the robot 10 by viewing the display screen800.

Further, the display screen 800 includes an image 810 indicating thedirection in which the operator is looking, such as the display positionof the spherical image displayed on the display terminal 50 or theimaging position of the generic image capturing device 24. For example,the image 810 includes a long arrow 811 indicating the display directionof the spherical image, and a short arrow 813 indicating the imagingdirection of the generic image capturing device 24. This enables theuser at the site where the robot 10 is located to recognize whichdirection the operator is looking on the screen displayed on the displayterminal 50. For example, even when the operator is facing the front onthe display screen 800, there is a case in which the operator is lookingbehind or beside the robot 10 on the spherical image displayed on thedisplay terminal 50. Therefore, the user at the site where the robot 10is located can recognizes the direction in which the operator is lookingby viewing the image 810 displayed on the display screen 800.

A display screen 850 illustrated in FIG. 43B displays an imagesimulating the back of the operator's head instead of the image of theoperator's face. The display screen 850 is displayed on the display 150b provided at the rear side of the robot 10 b 2 as illustrated in FIG.10B, for example. In this case, the display screen 800 described aboveis displayed on display 150 a provided at the front side of the robot 10b 2. Further, the display screen 850 also includes the image 810indicating the direction in which the operator is looking, insubstantially the same manner as the display screen 800. This enablesthe user who is around the robot 10 to recognize which direction theoperator is looking even from behind the robot 10. The image simulatingthe back of the operator's head is just one example of an imagedisplayed on the display screen 850. In another example, an imageobtained by actually photographing the back of the operator's head isdisplayed on the display terminal 50.

In the embodiment, the display direction of the spherical image (thearrow 811) corresponds to the “eyes” of the operator, and the imagingdirection of the generic image capturing device 24 (the arrow 813)corresponds to the “neck or head” of the operator. The “eyes” of theoperator represent the direction in which the operator is glancing. The“neck or head” of the operator represents the direction in which theoperator is looking closely. Around the robot 10, there are a directionin which the operator is looking closely and a direction in which theoperator is glancing. By displaying the arrow 811 and the arrow 813 onthe display screen 800 a and the display screen 800 b, the robot 10enables the user around the robot 10 to accurately recognize thedirection in which the operator is looking and to feel the intention ofthe operator.

In another example, the image 810 includes, instead of the arrow 811 andthe arrow 813, points whose positions indicate the directions. In stillanother example, the image 810 includes, instead of the arrow 811 andthe arrow 813, an icon(s) of an eye and/or a neck, so that the user canintuitively recognize the directions. In still another example, therobot 10 informs the user around the robot 10 of the direction in whichthe operator is looking by using a notification device such as theindicator lamps 160, as the robot 10 c 3 illustrated in FIG. 10C.

Variations of Embodiment:

A description is now given of a remote control system 1 b, according toa variation of the embodiment. In the variation, the same referencenumbers are allocated to elements (members or components) having thesame function as those of the above-described embodiment, and redundantdescription thereof is omitted below. In the remote control system 1 baccording to the variation of the embodiment, an information processingserver 70 generates a display image to be displayed on the displayterminal 50.

System Configuration:

FIG. 44 is a diagram illustrating an example of a system configurationof the remote control system 1 b, according to a variation of theembodiment. As illustrated in FIG. 44, the remote control system 1 baccording to the variation of the embodiment includes the informationprocessing server 70 in addition to the configuration illustrated inFIG. 1. The information processing server 70 is communicably connectedto the robots 10, the display terminal 50, and the communicationmanagement server 90 through a communication network 9. The informationprocessing server 70 exchanges image data with the robot 10 or thedisplay terminal 50 using a communication session established by thecommunication management server 90. The information processing server 70performs image processing on spherical image data and detailed imagedata transmitted from the robot 10 according to a state of the robot 10,and transmits the processed data to the display terminal 50. Theinformation processing server 70 is an example of an output controlapparatus. In one example, the information processing server 70 isconfigured as a single computer. In another example, the informationprocessing server 70 is configured as a plurality of computers to whichdivided units (functions, means, or storages) are arbitrarily allocated.

In this variation, the information processing server 70 and thecommunication management server 90 constitute a server system 7. Inanother example, the server system 7 is configured as a single computerincluding the units (functions or means) of the information processingserver 70 and the communication management server 90. In addition, therobot 10 and the server system 7 constitute a site control system 3.Further, the display terminal 50 and the server system 7 constitute adisplay control system 5.

Since the hardware configuration of the information processing server 70is the same or substantially the same as the hardware configuration ofthe communication management server 90 as illustrated in FIG. 13, aredundant description thereof is omitted below. In the followingdescription, it is assumed that the information processing server 70 hasthe hardware configuration as illustrated in FIG. 13.

Functional Configuration:

FIG. 45 is a diagram illustrating an example of a functionalconfiguration of the remote control system 1 b, according to a variationof the embodiment. Since the functions of apparatuses, terminals, andservers other than the information processing server 70 are the same orsubstantially the same as the functions as illustrated in FIG. 20,redundant descriptions thereof are omitted below.

The information processing server 70 includes a data exchange unit 71, adetermination unit 72, a data processing unit 73, and storage/readingunit 79. These units are functions that are implemented by or that arecaused to function by operating any of the elements illustrated in FIG.13 in cooperation with instructions of the CPU 901 according to thecontrol program expanded to the RAM 903. The information processingserver 70 further includes a storage unit 7000, which is implemented bythe ROM 902, the HD 904, or the storage medium 906 illustrated in FIG.13.

The data exchange unit 71, which is implemented by instructions of theCPU 901 and by the network I/F 908 illustrated in FIG. 13, has afunction of transmitting and receiving various types of data orinformation to and from other apparatuses through the communicationnetwork 9. For example, the data exchange unit 71 receives sphericalimage data and detailed image data from the robot 10 (control device 30)through the communication network 9. Further, for example, the dataexchange unit 71 receives the state information 170 indicating the stateof the robot 10 from the robot 10 (control device 30) through thecommunication network 9. Further, for example, the data exchange unit 71transmits display screen data generated by the data processing unit 73to the display terminal 50 through the communication network 9 to outputthe display screen data.

The determination unit 72, which is implemented by instructions of theCPU 901 illustrated in FIG. 13, has a function of making variousdeterminations. The data processing unit 73 is implemented byinstructions of the CPU 901 illustrated in FIG. 13. The data processingunit 73 has a function of generating display screen data to be displayedby the display terminal 50, based on various image data received by thedata exchange unit 71. Further, the data processing unit 73 has afunction of generating a request command, which is an execution requestfor causing the robot 10 to execute a particular processing.

The storage/reading unit 79, which is implemented by instructions of theCPU 901 illustrated in FIG. 13, has a function of storing various datain the storage unit 7000, and reads various data from the storage unit7000. In the storage unit 7000, a state information management DB 7001,a condition information management DB 7002, a user command management DB7003, and a display state management DB 7004 are stored. Since the stateinformation management DB 7001, the condition information management DB7002, the user command management DB 7003, and the display statemanagement DB 7004 have the same or substantially the sameconfigurations as the state information management DB 5001, thecondition information management DB 5002, the user command management DB5003, and the display state management DB 5004 of the display terminal50, respectively, redundant descriptions thereof are omitted below. Thestorage unit 7000 further stores spherical image data and detailed imagedata received by the data exchange unit 71. In one example, a sphericalimage data item and a detailed image data item stored in the storageunit 7000 are deleted when a predetermined time period has elapsed sincethe data items are received by the data exchange unit 71. In anotherexample, a spherical image data item and a detailed image data itemstored in the storage unit 7000 are deleted when a predetermined timeperiod has elapsed since the data items are transmitted (output) to thedisplay terminal 50 by the data exchange unit 71.

Processes or Operation of Variation of Embodiment:

A description is now given of processes or operation performed by theremote control system 1 b, according to a variation of the embodiment,with reference to FIG. 46 and FIG. 47. In the following descriptionprovided with reference to FIG. 46 and FIG. 47, processes performed bythe control device 30 of the robot 10 are described as processesperformed by the robot 10. FIG. 46 is a sequence diagram illustrating anexample of operation of transmitting various data from the robot 10 tothe display terminal 50 in the remote control system 1 b, according tothe variation of the embodiment.

First, the data exchange unit 31 of the robot 10 transmits, to theinformation processing server 70, spherical image data acquired by thespecial image capturing device 21 and detailed image data acquired bythe generic image capturing device 24 (step S201-1, step S201-2).Thereby, the data exchange unit 71 of the information processing server70 receives the spherical image data and the detailed image datatransmitted from the robot 10.

Next, the data processing unit 73 of the information processing server70 generates display screen data representing a display screen to bedisplayed by the display terminal 50, by using the spherical image dataand the detailed image data received by the data exchange unit 71 (stepS202). The display screen represented by the display screen datagenerated by the data processing unit 73 is, for example, the displayscreen 600 c as illustrated in FIG. 39A. In another example, the displayscreen represented by the display screen data generated by the dataprocessing unit 73 is the display screen 600 a illustrated in FIG. 30 orthe display screen 600 b illustrated in FIG. 32. Then, the data exchangeunit 71 of the information processing server 70 transmits the displayscreen data generated by the data processing unit 73 to the displayterminal 50 (step S203). Thereby, the data exchange unit 51 of thedisplay terminal 50 receives the display screen data transmitted fromthe information processing server 70.

Then, the display control unit 54 controls the display 511 to displaythe display screen 600 c represented by the display screen data receivedin step S203 (step S204). This enables the operator, who operates therobot 10 using the display terminal 50, to check a situation of the sitewhere the robot 10 is located while viewing the display screen 600 c onwhich the spherical image is displayed.

Next, the robot 10 changes the imaging position of the generic imagecapturing device 24 based on a request command or the like transmittedfrom the display terminal 50 (step S205). The state informationgeneration unit 35 generates the state information 170 indicating thestate of the robot 10 based on the drive state of the movement mechanism17 acquired from the movement control unit 41 (step S206). The processesof step S205 and step S206 are performed in the same or substantiallythe same manner as step S83 and step S84 described above with referenceto FIG. 38, and therefore redundant descriptions thereof are omittedbelow.

Then, the data exchange unit 31 transmits the state information 170generated by the state information generation unit 35 and the detailedimage data acquired by the captured image acquisition unit 38 to theinformation processing server 70 (step S207-1, step S207-2). Thereby,the data exchange unit 71 of the information processing server 70receives the state information 170 and the detailed image data. The dataprocessing unit 73 of the information processing server 70 performsimage processing on the display screen data generated in step S202 basedon the state information 170 and the detailed image data received by thedata exchange unit 71 (step S208). A description is now given ofcontents of the image processing performed by the information processingserver 70. FIG. 47 is a flowchart illustrating an example of anoperation of performing image processing on the display screen data,performed by the information processing server 70, according to thevariation of the embodiment.

When the data exchange unit 71 receives the state information 170transmitted from the robot 10 (YES in step S208-1), the operationproceeds to step S208-2. By contrast, when the data exchange unit 71receive no state information 170 (NO in step S208-1), the process ofstep S208-1 is repeated.

The storage/reading unit 79 reads the condition information stored inthe condition information management DB 7002 (step S208-2). Next, thedetermination unit 72 searches the condition information read by thestorage/reading unit 79, to determine whether there is any itemsatisfying the conditions indicated in the condition information amongthe variables indicated in the state information 170 received in stepS207-2 (step S208-3 a). Then, when the determination unit 72 determinesthat there is any item satisfying the condition indicated in thecondition information among the variables indicated in the stateinformation 170 (YES in step S208-3), the operation proceeds to stepS208-4. For example, when the imaging position of the generic imagecapturing device 24 indicated by the numerical values (parameters) ofthe variable names “CAMERA_H_ANGLE”, “CAMERA_V_ANGLE” and “CAMERA_ZOOM”included in the state information 170 is close to the display positionof the spherical image, the determination unit 72 determines that thecorresponding condition is satisfied. By contrast, when thedetermination unit 72 determines that there is no item satisfying thecondition indicated in the condition information among the variablesindicated in the state information 170 (NO in step S208-3), theoperation ends.

Next, the data processing unit 73 performs processing corresponding tothe item identified as satisfying the condition in step S208-3 on thedisplay screen data generated in step S202 (step S208-4). For example,when the imaging position of the generic image capturing device 24 isclose to the display position of the spherical image, the dataprocessing unit 73 generates display screen data representing thedisplay screen (600 d) displaying a detailed image represented by thedetailed image data received in step S207-2.

Referring again to FIG. 46, the data exchange unit 71 of the informationprocessing server 70 transmits the state information 170 received by thedata exchange unit 71 and the display screen data generated by the dataprocessing unit 73 to the display terminal 50 (step S209). Thereby, thedata exchange unit 51 of the display terminal 50 receives the stateinformation 170 and the display screen data, both being transmitted fromthe information processing server 70.

The storage/reading unit 59 of the display terminal 50 stores the stateinformation 170 received by the data exchange unit 51 in the stateinformation management DB 5001 (see FIG. 22A) stored in the storage unit5000 (step S210). Specifically, the storage/reading unit 59 stores eachof the numerical values included in the received state information 170in a field associated with the corresponding variable name of the stateinformation management table, thereby updating the value of each itemincluded in the state information management table. Then, the displaycontrol unit 54 controls the display 511 to display the display screen600 d represented by the display screen data received in step S209 (stepS211).

As described heretofore, even when the information processing server 70is used, the remote control system 1 b according to the variation of theembodiment controls the display terminal 50 to switch the displaybetween the spherical image and the detailed image according to thecondition. This allows the user to check the state of the remote siteefficiently and with high accuracy, thereby improving operability forthe user.

As described heretofore, an output control apparatus (e.g., the displayterminal 50 or the information processing server 70) according to anembodiment of the present disclosure is communicable with the controldevice 30 (an example of a communication apparatus) through thecommunication network 9, the control device 30 including the specialimage capturing device 21 (an example of first image capturing means)configured to capture an image of a subject at a remote site to acquirea spherical image (an example of a first image) and the generic imagecapturing device 24 (an example of second image capturing means)configured to capture a part of the subject to acquire a detailed image(an example of a second image). The output control apparatus receivesthe spherical image transmitted from the control device 30, outputs thereceived spherical image so as to be displayed on the display 511 (anexample of a display unit), and receives the detailed image, which isobtained by capturing an image of a part of the subject corresponding tothe display position of the spherical image displayed on the display511, from the control device 30. Further, the output control apparatusoutputs the received detailed image so as to be displayed on the display511, and controls the display 511 to display the output spherical imageand detailed image. Thus, the output control apparatus according to anembodiment of the present disclosure controls the display position ofthe spherical image and the imaging position of the generic imagecapturing device 24 to link with each other, thereby enabling the userto seamlessly recognize the entire remote site and an area where theuser has interest to details. Therefore, the output control apparatusimproves both the operability and the resolution of the display image,while displaying images acquired by different types of imaging means ina suitable manner for viewing by a user.

Further, when the display position of the spherical image (an example ofthe first image) displayed on the display 511 (an example of the displayunit) is changed, the output control apparatus (e.g., the displayterminal 50 or the information processing server 70) according to anembodiment of the present disclosure receives, from the control device30 (an example of the communication apparatus), the detailed image (anexample of the second image), which is obtained by capturing an image ofa part of the subject corresponding to the changed display position.Thus, the output control apparatus according to an embodiment of thepresent disclosure switch the display between the spherical image andthe detailed image according to the condition, thereby allowing a userto check the state of the remote site efficiently and with highaccuracy. This improves the operability for the user.

Furthermore, the output control apparatus (e.g., the display terminal 50or the information processing server 70) according to an embodiment ofthe present disclosure outputs the received detailed image (an exampleof the second image) when the display position of the spherical image(an example of the first image) and the imaging position of the genericimage capturing device 24 (an example of the second image capturingmeans) are within a predetermined range, and outputs the receivedspherical image when the display position of the spherical image and theimaging position of the generic image capturing device 24 are outsidethe predetermined range. Thus, the output control apparatus according toan embodiment of the present disclosure enables the user to view thespherical image to recognize the remote site until the imaging positionof the detailed image catches up to the display position of thespherical image, and to view the detailed image to recognize moredetailed information of the remote site when the imaging position of thedetailed image catches up to the display position of the sphericalimage. This improves operability for the user.

Still further, when the display position of the spherical image (anexample of the first image) is changed to a position that cannot beimaged by the generic image capturing device 24 (an example of thesecond image capturing means), the output control apparatus (e.g., thedisplay terminal 50 or the information processing server 70) accordingto an embodiment of the present disclosure generates the display screen600 e in which the image 655 (an example of an unacquirable image)indicating that the detailed image (an example of the second image)cannot be acquired is superimposed on the spherical image, and outputsthe generated display screen 600 e so as to be displayed on the display511 (an example of the display unit). Thus, when the display position ofthe spherical image is changed to a position that cannot be imaged bythe generic image capturing device 24, the output control apparatusaccording to an embodiment of the present disclosure allows the user tointuitively recognize a situation where the spherical image cannot beswitched to the detailed image.

Still further, the output control apparatus (e.g., the display terminal50 or the information processing server 70) according to an embodimentof the present disclosure displays a display screen (e.g., the displayscreen 600 b, the display screen 600 c, or the display screen 600 f) inwhich the imaging position icon 653 (an example of an imaging positionimage) indicating the imaging position by the generic image capturingdevice 24 (an example of the second image capturing means) issuperimposed on the spherical image (an example of the first image).Thus, the user recognizes the imaging position of the generic imagecapturing device 24 by viewing the imaging position icon 653 displayedon the display screen until the imaging position by the generic imagecapturing device 24 catches up to the display position of the sphericalimage. This enables the user to predict the timing when the sphericalimage is switched to the detailed image.

Still further, the output control apparatus (e.g., the display terminal50 or the information processing server 70) according to an embodimentof the present disclosure, generates a display screen (e.g., the displayscreen 600 b, the display screen 600 c, the display screen 600 e, or thedisplay screen 600 f) in which the drive direction icon 651 b (anexample of a drive direction image) indicating the drive direction ofthe robot 10 (an example of a mobile object) located at a remote site issuperimposed on the spherical image (an example of a wide-angle image).Thus, even when the user changes the display position (e.g., theviewable area T) of the spherical image by an operation of changing thedisplay direction of the spherical image or enlarging or reducing thespherical image, the drive direction icon 651 b is displayed on thedisplay screen. This enables the user to recognize the state (directionor drive direction) of the robot 10.

Still further, in the output control apparatus (e.g., the displayterminal 50 or the information processing server 70) according to anembodiment of the present disclosure, the spherical image (an example ofthe wide-angle image) is represented as a solid sphere (e.g., the solidsphere CS) in a three-dimensional virtual space. The output controlapparatus arranges the object 651 a (an example of an object image)indicating the position of the robot 10 (an example of a mobile object)located at a remote site at the center position of the three-dimensionalvirtual space, and generates, as a display screen (e.g., the displayscreen 600 b, the display screen 600 c, the display screen 600 e, or thedisplay screen 600 f), an image viewed from a virtual camera IC providedat a position of a viewpoint from which the spherical image is viewed,the virtual camera being arranged upward and rearward from the centerposition. Thus, the output control apparatus according to an embodimentof the present disclosure arranges the virtual camera IC at a positionapart from the center position of the three-dimensional virtual space(solid sphere CS) having a spherical shape, and sets the position wherethe virtual camera IC is arranged as the origin of the field of view,thereby bringing the existence of the robot 10 (the object 651 a) withinthe field of view of the virtual camera IC. Further, this enables theuser of the display terminal 50 to recognize both the existence of therobot 10 (the object 651 a) and the spherical image (e.g., theviewable-area image Q) at the same time from a so-called pseudothird-person view, thereby enabling the user to recognize the situationof the site smoothly and improving operability of the robot 10.

Still further, when the display position of the spherical image (anexample of the first image) is changed, the output control apparatus(e.g., the display terminal 50 or the information processing server 70)according to an embodiment of the present disclosure generates an imageviewed from the virtual camera IC whose position is changed so that theobject 651 a (an example of the object image) is viewable as a displayscreen (e.g., the display screen 600 b, the display screen 600 c, thedisplay screen 600 e, or the display screen 600 f). Thus, the outputcontrol apparatus according to an embodiment of the present disclosureenables the user to recognize the presence of the robot (the object 651a), even when the display position (e.g., the viewable area T) of thespherical image is changed.

Still further, the output control apparatus according to an embodimentof the present disclosure is the display terminal 50 communicable withthe control device 30 (an example of the communication apparatus)through the communication network 9. The display terminal 50 displaysthe received spherical image (an example of the first image) on thedisplay 511 (an example of the display unit), changes the displayposition of the displayed spherical image, and transmits, to the controldevice 30, a change request for requesting to change the imagingposition of the generic image capturing device 24 (an example of thesecond image capturing means) to the changed display position. Further,the display terminal 50 receives the detailed image (an example of thesecond image) that is acquired by the generic image capturing device 24in response to the transmitted change request, and displays the receiveddetailed image on the display 511 together with the spherical image.Thus, the display terminal 50 controls the display position of thespherical image and the imaging position of the generic image capturingdevice 24 to link with each other, thereby enabling the user toseamlessly recognize the entire remote site and an area where the userhas interest to details. Therefore, the display terminal 50 improvesboth the operability and the resolution of the display image, whiledisplaying images acquired by different types of imaging means in asuitable manner for viewing by the user.

Moreover, when the display position of the spherical image (an exampleof the first image) and the imaging position of the generic imagecapturing device 24 (an example of the second image capturing unit) arewithin a predetermined range, the display terminal 50 according to anembodiment of the present disclosure displays the acquired detailedimage (an example of the second image) on the display 511 (an example ofthe display unit), and when the display position of the spherical imageand the imaging position of the generic image capturing device 24 areoutside the predetermined range, the display terminal 50 according to anembodiment of the present disclosure displays the acquired sphericalimage on the display 511. Thus, the display terminal 50 enables the userto view the spherical image to recognize the remote site until theimaging position of the detailed image catches up to the displayposition of the spherical image, and to view the detailed image torecognize more detailed information of the remote site when the imagingposition of the detailed image catches up to the display position of thespherical image. This improves operability for the user.

Furthermore, the robot 10 according to an embodiment of the presentdisclosure is a mobile object communicable with the output controlapparatus (e.g., the display terminal 50 or the information processingserver 70) through the communication network 9, and includes the controldevice 30 (an example of the communication apparatus), the movementmechanism 17 that moves the robot 10, and plural generic image capturingdevices 24 (an example of the second image capturing means) whoseimaging direction is changeable. The robot 10 transmits the detailedimage (an example of the second image) acquired by any of the pluralgeneric image capturing devices 24 to the output control apparatus. Thisenables the robot 10 to reduce blind spots such as the ceiling or thefloor in the image capturing by the generic image capturing device 24.

Still further, the robot 10 (an example of the mobile object) accordingto an embodiment of the present disclosure includes display means (e.g.,the display 150 or the indicator lamps 160) configured to indicate thedirection in which the user is looking at the spherical image (anexample of the first image). Still further, the robot 10 receives acaptured image obtained by capturing the user from the output controlapparatus (e.g., the display terminal 50 or the information processingserver 70), and the display 150 (an example of the display unit)displays the received captured image (e.g., the display screen 800).Still further, at least one of the indicator lamps 160 of the robot 10is turned on indicating the direction in which the user of the displayterminal 50 is looking at the spherical image. Thus, the robot 10enables the user who is at the site where the robot 10 is located torecognize who is operating the robot 10 and the direction the operatoris looking at.

Further, the display terminal 50 according to an embodiment of thepresent disclosure performs streaming communication with the controldevice 30 (an example of the communication apparatus) including thespecial image capturing device 21 (an example of image capturing means)configured to capture an image of a subject at a remote site, tocommunicate the spherical image (an example of a wide-angle image)captured by the special image capturing device 21. The display terminal50 includes the display control unit 54 (an example of display controlmeans) that controls the display 511 (an example of the display unit) todisplay a predetermined area (e.g., the viewable area T), which is apart of the spherical image transmitted from the control device 30, andthe mouse 500 (an example of external input means) that is connected tothe display terminal 50 and controls the display of the spherical imagedisplayed on the display 511. The display control unit 54 changes thepredetermined area, which is a part of the spherical image, to bedisplayed on the display 511 according to the movement of the mouse 500.Thus, the display terminal 50 changes an area, which is a part of thespherical image, to be displayed on the display 511 according to themovement of the mouse 500, thereby improving operability when the userchecks the situation of the remote site while viewing the sphericalimage.

Still further, the display terminal 50 according to an embodiment of thepresent disclosure further includes the acceptance unit 52 (an exampleof detection means) configured to detect a movement of the mouse 500 (anexample of the external input means). The display control unit 54 (anexample of the display control means) of the display terminal 50controls the display 511 (an example of the display unit) to display afirst area, which is a part of the spherical image (an example of thewide-angle image), and in response to detection of the movement of themouse 500, controls the display 511 to display a second area, which is apart of the spherical image and an area different from the first area.Thus, the display terminal 50 controls the display 511 to display animage indicating the different area, which is a part of the sphericalimage, according to the movement of the mouse 500, thereby improvingoperability when the user checks the situation of the remote site whileviewing the spherical image.

Still further, the display terminal 50 according to an embodiment of thepresent disclosure further includes a display position identificationunit 55 (an example of calculation means) configured to calculate themovement amount of the mouse 500 (an example of the external inputmeans). The display control unit 54 (an example of the display controlmeans) of the display terminal 50 controls the display 511 to displaythe second area, which is away from the center of the first areadisplayed on the display 511 by a distance corresponding to thecalculated movement amount. Thus, the display terminal 50 changes thedisplay position of the spherical image according to the movement amountof the mouse 500 instead of the movement amount of the mouse cursor,thereby smoothly displaying an image of the position that the user wantsto view independent of the screen size or the like of the display 511.

The related art does not mention how to output images acquired bydifferent types of cameras (imaging means). Thus, depending on how theimages are output, there is a drawback that a user may not checkinformation on a remote site in the user's desired manner.

According to one or more embodiments of the present disclosure, imagesacquired by different types of imaging device are displayed in asuitable manner for viewing by a user.

Supplementary Information on Embodiments:

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), digital signal processor (DSP), fieldprogrammable gate array (FPGA), and conventional circuit componentsarranged to perform the recited functions.

Further, the various tables of the embodiment described above may begenerated by the learning effect of machine learning. By classifyingdata of the associated items, the use of the tables is optional. In thepresent disclosure, machine learning is a technique that enables acomputer to acquire human-like learning ability. Machine learning refersto a technology in which a computer autonomously generates an algorithmrequired for determination such as data identification from learningdata loaded in advance, and applies the generated algorithm to new datato make a prediction. Any suitable learning method is applied formachine learning, for example, any one of supervised learning,unsupervised learning, semi-supervised learning, reinforcement learning,and deep learning, or a combination of two or more those learning.

Although the output control apparatus, the display terminal, the remotecontrol system, the control method, and the program according toembodiments of the present disclosure are described above, theabove-described embodiments are illustrative and do not limit thepresent disclosure. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present disclosure. Any one of the above-describedoperations may be performed in various other ways, for example, in anorder different from the one described above.

What is claimed is:
 1. An output control apparatus communicable with acommunication apparatus through a communication network, thecommunication apparatus including a first image capturing deviceconfigured to capture a subject at a remote site to acquire a firstimage and a second image capturing device configure to capture a part ofthe subject to acquire a second image, the output control apparatuscomprising circuitry to: receive the first image transmitted from thecommunication apparatus; output the received first image so as to bedisplayed on a display; receive, from the communication apparatus, thesecond image acquired by capturing a part of the subject correspondingto a display position of the first image displayed on the display;output the received second image so as to be displayed on the display;and control the display to display the first image and the second imagethat are output.
 2. The output control apparatus of claim 1, whereinwhen the display position of the first image displayed on the display ischanged, the circuitry receives, from the communication apparatus, thesecond image acquired by capturing a part of the subject correspondingto the changed display position.
 3. The output control apparatus ofclaim 1, wherein when the display position of the first image and animaging position by the second image capturing device are within apredetermined range, the circuitry outputs the second image received,and when the display position of the first image and the imagingposition by the second image capturing device are outside thepredetermined range, the circuitry outputs the first image received. 4.The output control apparatus of claim 1, wherein when the displayposition of the first image is changed to a position that cannot beimaged by the second image capturing device, the circuitry outputs adisplay screen in which an unacquirable image indicating that the secondimage cannot be acquired is superimposed on the first image.
 5. Theoutput control apparatus of claim 1, wherein the circuitry is furtherconfigured to output a display screen in which an imaging position imageindicating an imaging position by the second image capturing device issuperimposed on the first image.
 6. The output control apparatus ofclaim 1, wherein the communication apparatus is provided in a mobileobject located at the remote site, and the circuitry is furtherconfigured to output a display screen in which a drive direction imageindicating a drive direction of the mobile object is superimposed on thefirst image.
 7. The output control apparatus of claim 6, wherein thefirst image is represented as a solid sphere in a three-dimensionalvirtual space, and the circuitry is further configured to arrange anobject image indicating a position of the mobile object at a centerposition of the three-dimensional virtual space; and output an imageviewed from a virtual camera located at a position of a viewpoint forviewing the first image, the virtual camera being arranged at a positionaway from the center position.
 8. The output control apparatus of claim7, wherein the circuitry outputs, as the display screen, an image viewedfrom the virtual camera that is arranged behind the center position sothat the object image is visible.
 9. The output control apparatus ofclaim 8, wherein when the display position of the first image ischanged, the circuitry outputs, as the display screen, an image viewedfrom the virtual camera whose position is changed so that the objectimage is visible.
 10. A display terminal constituted as the outputcontrol apparatus of claim 1, communicable with the communicationapparatus through a communication network, the display terminalcomprising circuitry configured to: controlling the display to displaythe first image received by the circuitry; change the display positionof the displayed first image; transmit, to the communication apparatus,a change request for requesting to change an imaging position of thesecond image capturing device to the changed display position; receivethe second image that is acquired by the second image capturing devicein response to the transmitted change request; and control the displayto display the received second image together with the first image. 11.The display terminal of claim 10, wherein when the display position ofthe first image and the imaging position by the second image capturingdevice are within a predetermined range, the circuitry controls thedisplay to display the acquired second image, and when the displayposition of the first image and the imaging position by the second imagecapturing device are outside the predetermined range, the circuitrycontrols the display to display the acquired first image.
 12. Thedisplay terminal of claim 10, further comprising: an external inputdevice connected to the display terminal, the external input devicecontrolling the display position of the first image displayed on thedisplay, wherein the circuitry changes the display position of thedisplayed first image according to movement of the external inputdevice.
 13. A remote control system comprising: the output controlapparatus of claim 1; and a mobile object communicable with the outputcontrol apparatus through a communication network.
 14. A control methodperformed by an output control apparatus communicable with acommunication apparatus through a communication network, thecommunication apparatus including a first image capturing deviceconfigured to capture a subject at a remote site to acquire a firstimage and a second image capturing device configured to capture a partof the subject to acquire a second image, the control method comprising:receiving the first image transmitted from the communication apparatus;outputting the received first image so as to be displayed on a display;and receiving, from the communication apparatus, the second imageacquired by capturing a part of the subject corresponding to a displayposition of the first image displayed on the display, wherein theoutputting includes outputting the received second image so as to bedisplayed on the display, and the method further comprising controllingthe display to display the first image and the second image that areoutput.
 15. A non-transitory computer-readable medium storing a programthat causes a computer to execute the method of claim 14.